Greetings! We are Rakesh Mohan, Director, and Lance McCleve, Principal Evaluator, of the Idaho legislature’s Office of Performance Evaluations. Recognizing the ongoing debate about evaluator credentialing/certification, here we discuss a study that demonstrates the value of credentials in evaluations but not necessarily credentialed evaluators.
As the director, I manage high stakes evaluations. A poorly conducted evaluation could end my career and damage my office’s hard-earned credibility. For me, managing evaluations involves assembling the best team with the resources given.
Hot Tip:
The best team collectively has the total knowledge necessary to conduct and disseminate a credible evaluation. It includes evaluative thinking, ability to conceptualize complex issues, subject matter expertise, project management experience, research and analytical skills, political savvy, knowledge of evaluation standards, and communication skills. Depending on the type of evaluation, the team may benefit from having certain credentialed members.
Lessons Learned:
In 2008 the Idaho governor proposed a fuel tax to increase highway funding, but many legislators were not interested in raising taxes for a department that they thought was poorly managed. Through a resolution, the legislature asked us to evaluate the transportation department – a highly political assignment. By accepting this assignment, I took the biggest professional risk ever – I had no knowledge of transportation or engineering, the study had a big scope, and the report was due in six months.
I contracted with a team of 11 consultants who brought expertise in transportation, engineering, construction, systems analysis, procurement and contract administration, capital project management, performance auditing, and performance measurement. This team included three licensed professional engineers and a certified project management professional. Additionally, I hired one consultant for financial analysis and two to provide quality assurance and help me manage the politics involved. Interestingly, no one on the team had the title of evaluator; the closest title was management auditor.
The evaluation was a success—on time and under budget! We identified $30.6 million in potential savings the first five years and an additional $6.6 million annually thereafter contingent on implementation of our recommendations.
The governor issued two executive orders: one required the department to implement our recommendations and the other established a task force to identify alternative sources of highway funding. The legislature passed bills raising $27.7 million for highways and appropriated about $10 million to implement the recommended asset management systems.
Stakeholders found the evaluation useful because all parties saw it as accurate, thorough, and unbiased. They knew that I had hired highly qualified consultants to work on the evaluation. Clearly credentials have a place in evaluation but whether evaluators should be credentialed is not so clear. Hence, the discussion about the value of credentialing evaluators should continue.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Thank you Talbot, you summed up my reservations superbly. On one hand, I would like to see the ‘professionalisation’ of evaluation, but credentialing is no guarantee of quality – we all know of incompetent surgeons & lawyers. If NZ does ever decide to have credentialing – not likely in the near future, I think- I would probably apply. Meantime, I will follow the discussion with interest.
Hello April,
Thank you for chiming in. We are thrilled with the amount of discussion that has happened with this post.
The main purpose of our post was to emphasize the importance of assembling an evaluation team with the “total knowledge” necessary to conduct and disseminate a credible evaluation.
We certainly do not think the issue of credentialing is straightforward; it is in need of a healthy, ongoing discussion. Glad to hear you will be following the discussion and we hope you continue to contribute.
Best wishes,
Rakesh and Lance
This was a very insightful blog and made me think again about all my evaluation colleagues and the expertise they bring to evaluation work. Credentialing is a very slippery slope, costly and often fraught with politics and error (as in evaluation error)
Hello Ann,
Thank you for responding to our post. As the director of a small office, I’m fortunate to have a great group of staff with many different skills and an access to a number of highly skilled consultants.
We are pleased that so much effort is being made around the world to professionalize the field of evaluation as shown in two recent journal volumes:
1. Accreditation, Certification, and Credentialing: Relevant Concerns for U.S. Evaluators. New Directions for Evaluation, Number 145, Spring 2015.
2. The Canadian Journal of Program Evaluation, Volume 29, Number 3, 2015.
All the best,
Rakesh
Hi Rickesh,
I just submitted my Credentialed Evaluator (CE) application from the CES, and would like to share some insights about the process and my reflections on your post.
You note that no one on the team had the title of evaluator, but I believe you would call yourself an evaluator. As an evaluator, you recognized the limits of your own skills and knowledge and put together a team that met the requirements of the project. That is one of the many competencies you need to demonstrate to become a credential evaluator.
It is true that having a CE does not mean that you are qualified to work on any evaluation. For instance, I know credentialed evaluators that practice from a primarily qualitative framework. I consider them very qualified evaluators, but would not suggest that they lead projects needing lots of advanced statistics.
While a CE does not guarantee that an evaluator is the right person for the job, I do think it is helpful to demonstrate a certain level of expertise and commitment to standards of practice.
Hope that helps,
Paul
Hi Paul,
Congratulations on submitting your application for Credentialed Evaluator.
Thank you much for responding to our post. Your perspective on credentials is very important because you just submitted your application.
We are not trying to comment on the value of credentialed evaluators in general. We agree with you that credentials “demonstrate a certain level of expertise and commitment to standards of practice.” That is why I decided to assembled my team with certified and licensed professionals. In addition to demonstrating a certain level of expertise, I was also interested in improving the perceived credibility of our evaluation. It was a high stakes evaluation that was conducted in a politically charged environment.
The perception of credibility is important to foster evaluation use, especially in highly political environments. In the context of our example, credentials of the evaluator would have done little to improve the perception of credibility and the evaluation would have likely been ignored. However, having three licensed professional engineers (PE) made all the difference.
We appreciate your commitment to showcase the value of high standards of practice for our profession. Best wishes,
Rakesh
Hi Rakesh,
Yes, currently an CE designation improves the perception of credibility in a minority of instances. Hopefully, as the Canadian project matures more clients will value having someone with a CE on the team to the same extent as they value having someone with say their professional engineer’s license. I have seen some RFPs call for the lead evaluator to have a CE, and I think the CE will become more important overtime, but it will take time.
Great to see the evaluation community is looking to learn for the Canadian experience.
Cheers,
Paul.
Thanks for your valuable contribution on this post, Paul.
I commend the Canadian Evaluation Society for embarking on this path — no doubt, it was a tough decision to make. We all have much learn from this Canadian experience. Best, Rakesh
Clearly this success story does not make the case for or against credentialing. The fact that evaluation is currently carried out mostly by non-evaluators is unsurprising. Evaluation is not yet a profession that controls access to the designation… More damaging to the credentialistas would be a story that describes a bad evaluation carried out by a credentialed evaluator. Is the lesson of this case that evaluation is a meta-discipline so that experienced evaluators such as Rakesh and Lance may have a comparative advantage putting together and managing multi-disciplinary assessment teams?
Bob
Bob, thanks for commenting and pointing out that this story does not make the case for or against credentialing.
We wanted to use this story to say that evaluators should be aware that stakeholders and evaluation sponsors (at least in the environment we work in) make judgments about the credibility of our evaluations. They make these judgments even if they have a limited understanding of what makes an evaluation high quality.
In the transportation example, I quickly learned during initial conversations with sponsors and stakeholders that they valued professional licensing/certification. Therefore, I thought that one of the best ways to foster the perception of evaluation credibility and ultimately use was by bringing in licensed/certified experts.
I won’t pretend that we had in mind your point about the lesson being that evaluation is a meta-discipline all along, but we certainly think that is a fair take-away. Again, thanks for participating in the discussion.
Rakesh and Lance
Rakesh and Lance,
This is great stuff! But what does it have to do with the value of credentials as indicated in the subject line? I can’t see any reference to this in the post, and fail to see how credentialing would have made your experience better (or worse). It seems that you purposely, and intelligently (vs. in a mechanistic manner) assessed and selected evaluation team members with required competencies – and reading between the lines, provided appropriate support and supervision. I fail to see how credentials would – or should – have done very much to aid your selection and project management approach.
Burt
Hi Burt,
It is so nice to hear from you. The credentials we are referring to are the credentials of the consultants selected to work on the team. I appreciate you saying I “purposely, and intelligently (vs. in a mechanistic manner) assessed and selected evaluation team members.”
As the director of the office, my primary concern was the competency of the team — both in fact and perceived. The Idaho legislature did not necessarily require me to have anyone on the team with formal professional credentials. However, because of the significant politics at play in the environment in which we conducted the study, I made a conscious decision to select consultants who were qualified and had credentials. The team included three licensed professional engineers (PE) and one certified project management professional.
We provided this evaluation as an example of credentials (although not evaluator credentials) improving the perceived credibility of the evaluation in certain situations. The absence of perceived credibility could jeopardize the evaluation use.
Hope to see you at the AEA conference in Chicago. All the best,
Rakesh
Good distinction. Drawing on credentialed sources is a different issue from being credentialed. I believe evaluation credentialing could be valuable in two specific situations: (1) Applying for an assignment from an evaluation-naive client who is worried about assessing applicants’ qualifications; (2) defending unwelcome evaluation results before disappointed clients who seek to deflect the findings by charging that the evaluator did not follow professional practice. Seeking new projects and defending results still require that the evaluator demonstrate knowledge and judgement. A credential might save some time and effort. This formulation of the issue implies that the ratio of effort saved to cost and effort of obtaining the credential would need to be >1 for the credentialing program to be successful.
Thank you Talbot for emphasizing the distinction in our post. It is hard to do that in the few words we are allowed in the post.
You described my concerns for the evaluation in our example quite well. I was worried that I would end up defending unwelcome evaluation results to disappointed clients who would be trying to deflect the findings by charging that the evaluators did not follow professional practice. Knowing our audience, I knew that drawing on licensed/credentialed sources was my best bet for avoiding the situation you described.
Thanks for participating in this discussion. Best,
Rakesh and Lance