The Program Evaluation Standards – Guidelines and Practitioner’s Guide by Thanos Patelis
Hello, everyone, my name is Thanos Patelis, and I am a director of test development and assessment at the Center for Talented Youth at Johns Hopkins University and lead psychometrician at the Center for Certification and Competency-Based Education at the University of Kansas. While I mostly do measurement work, I frequently undertake large-scale evaluations. Recently, I started to design an efficacy study of an assessment, feedback and occupational guidance system for well-suited individuals for entry-level occupations in cybersecurity. While the Program Evaluation Standards provide guidelines for program evaluations, I wanted to offer here how we can use the Program Evaluation Standards to guide the design, implementation, and dissemination of the study. The Program Evaluation Standards can guide practice!
Hot Tips
The Program Evaluation Standards are organized in five sets of guidelines involving utility (does the evaluation serve the information needs of the program), feasibility (does the evaluation take place effectively and efficiently), propriety (is the evaluation proper, fair, legal, right, and acceptable), accuracy (are the evaluation representations, propositions, and findings truthful), and evaluation accountability (are the evaluation results responsibly used to produce value).
As I launched the design of the efficacy research on this assessment-feedback-occupational guidance system, I started with the utility standards to ensure the goals and purposes of the evaluation were developed with the stakeholders as co-designers. Then, I jumped to the evaluation accountability standards to ensure agreement by both the evaluation team and the stakeholders involved the nature of the documentation and the use of the results. Next, I dove into the propriety standards to make sure agreements, letters of consent, contracts, and all documentation and practices of the evaluation represented fairness and legal requirements. Then, I jumped into the accuracy standards to make sure the intended efficacy research was designed to maximize the strength of our claims from the results. Finally, as I prepare to start the efficacy study, I attend to the feasibility standards to make sure proper management methods are used, and contextual information is included rather than removed.
I use the Program Evaluation Standards to know what makes for a quality evaluation. But with a different order than how the standards are presented in the publication, I use them to step through the design, implementation, and dissemination functions of the evaluation. In the order of utility, evaluation accountability, propriety, accuracy, and feasibility standards, the Program Evaluation Standards are my practitioner’s guide, as well.
This week, we’re diving the Program Evaluation Standards. Articles will (re)introduce you to the Standards and the Joint Committee on Standards for Educational Evaluation (JCSEE), the organization responsible for developing, reviewing, and approving evaluation standards in North America. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.