RoE TIG Week: Evaluator Competencies Assessment Tool (ECAT) by Minji Cho, Ann Marie Castleman, Haley Umans, and Mike Mwirigi

Hi! We are Minji Cho, Ann Marie Castleman, Haley Umans, and Mike Mwirigi. We are students in the Claremont Graduate University’s Evaluation and Applied Research doctoral program. For the past year, we have been developing the Evaluator Competencies Assessment Tool (ECAT) which provides evaluators and researchers with a survey to evaluate competencies in five domains: professional practice, methodology, context, planning and management, and interpersonal skills.

We developed the ECAT using the official AEA list of competencies adopted in 2018. We chose this source based on the thorough and credible process undertaken by the AEA Evaluator Competencies taskforce and the evaluation community to develop it. The survey modified the competencies into clear and concise survey items, based on best practices in survey development (see Mail and Internet Surveys: The Tailored Design Method by Don Dillman). The tool has a 7-point Likert scale, from 0 (novice) to 6 (expert).

We conducted two research studies to develop evidence for the structural and construct validity of the ECAT. Using confirmatory factor analysis, we found evidence that the competencies identified fit within the five previously established competency domains. Moreover, we found evidence for convergent validity; evaluators with more years of evaluation experience, more training, with graduate degrees, and those who view evaluation as their primary profession had higher competencies scores. Furthermore, we also found evidence for divergent validity given that evaluator’s gender and race were not related to differences in competencies scores.

Overall, one of the biggest takeaways to date is that there is no substitute for experience. The number of years an evaluator has been practicing evaluation is most predictive of competencies across all five domains.

Hot Tip: Evaluators might consider using the ECAT as a self-reflection tool to assess their strengths and weaknesses in one or more domains as they grow in their careers.

Hot Tip: In the future, researchers can use the survey as a measurement tool to look at domain specific or overall competencies among program evaluators.

The Evaluator Competencies Assessment Tool (ECAT) is available upon request (minji.cho@cgu.edu). Please reach out to learn more or access the tool!

---

The American Evaluation Association is celebrating RoE TIG Week with our colleagues in the Research on Evaluation TIG. All of the blog contributions this week come from our RoE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.