RoE TIG Week: Evaluator Competencies Assessment Tool (ECAT) by Minji Cho, Ann Marie Castleman, Haley Umans, and Mike Mwirigi

Hi! We are Minji Cho, Ann Marie Castleman, Haley Umans, and Mike Mwirigi. We are students in the Claremont Graduate University’s Evaluation and Applied Research doctoral program. For the past year, we have been developing the Evaluator Competencies Assessment Tool (ECAT) which provides evaluators and researchers with a survey to evaluate competencies in five domains: professional practice, methodology, context, planning and management, and interpersonal skills.

We developed the ECAT using the official AEA list of competencies adopted in 2018. We chose this source based on the thorough and credible process undertaken by the AEA Evaluator Competencies taskforce and the evaluation community to develop it. The survey modified the competencies into clear and concise survey items, based on best practices in survey development (see Mail and Internet Surveys: The Tailored Design Method by Don Dillman). The tool has a 7-point Likert scale, from 0 (novice) to 6 (expert).

We conducted two research studies to develop evidence for the structural and construct validity of the ECAT. Using confirmatory factor analysis, we found evidence that the competencies identified fit within the five previously established competency domains. Moreover, we found evidence for convergent validity; evaluators with more years of evaluation experience, more training, with graduate degrees, and those who view evaluation as their primary profession had higher competencies scores. Furthermore, we also found evidence for divergent validity given that evaluator’s gender and race were not related to differences in competencies scores.

Overall, one of the biggest takeaways to date is that there is no substitute for experience. The number of years an evaluator has been practicing evaluation is most predictive of competencies across all five domains.

Hot Tip: Evaluators might consider using the ECAT as a self-reflection tool to assess their strengths and weaknesses in one or more domains as they grow in their careers.

Hot Tip: In the future, researchers can use the survey as a measurement tool to look at domain specific or overall competencies among program evaluators.

The Evaluator Competencies Assessment Tool (ECAT) is available upon request (minji.cho@cgu.edu). Please reach out to learn more or access the tool!


The American Evaluation Association is celebrating RoE TIG Week with our colleagues in the Research on Evaluation TIG. All of the blog contributions this week come from our RoE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

1 thought on “RoE TIG Week: Evaluator Competencies Assessment Tool (ECAT) by Minji Cho, Ann Marie Castleman, Haley Umans, and Mike Mwirigi”

  1. Dr. Sondra LoRe

    How exciting!! Thank you for taking the initiative to develop such a valuable tool!
    I can envision so many ways in which this can support evaluation teams such as mine housed within a university, industry evaluators, consultants, and emerging evaluators. Understanding where we shine our light (to pull in the conference theme :)) goes a long way toward serving the people intended to be served in our evaluation work! Kudos to your team. Minji- I will be in touch with you via email!
    Sondra LoRe, Ph.D.
    Manager | National Institute for STEM Evaluation and Research (NISER)
    Adjunct Professor | Evaluation, Statistics, and Measurement Program, Department of Educational Psychology & Counseling

    The University of Tennessee, Knoxville
    Office of Research & Engagement
    114 Philander P. Claxton Education Building
    PH: 865-974-4962 | Fax: 865-974-9300 | https://www.stemeval.org
    slore@utk.edu

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.