MSI Fellowship Week: Smita Mehta On Asking the Right Questions: First Step toward Rigor in Program Evaluation Smita Mehta

Hello!  I am Smita Shukla Mehta, a 2016 Fellow of AEA’s Minority Serving Institutions program.  I am a Professor of Special Education in the Department of Educational Psychology at the University of North Texas.  I teach doctoral courses in program evaluation and feel passionate about providing graduate students opportunities to develop technical competencies in conceptualizing and conducting evaluations of specific programs.  I have been fortunate to have several funded grant projects through the U.S. Department of Education that require systematic evaluation; thus I get to practice what I teach!

Blog Topic:

The topic of this blog is Asking the Right Questions: First Step toward Rigor in Program Evaluation.  I chose this topic because rigor in program evaluation necessitates directly connecting the [right] questions about the program outcomes and utilization of the findings.  Whether the questions are about the process, outcomes or both, they should address the purpose for which we evaluate programs (e.g., accountability, improvement, knowledge development, and oversight).

Hot Tips:

1. Asking the right questions is not just an intellectual exercise to showcase our technical skills. It necessitates a deep and clear understanding of the program being evaluated, its institutional culture, its stage in the life-cycle, and expected impact for stakeholders.  Reflect on the importance of a comprehensive framework for evaluation.
2. Understand the program from the perspectives of the stakeholders – those who administer and/or implement the program, and those who benefit from the services. Engage stakeholders in the planning and evaluation process for credibility and accuracy.  Know the purpose of the evaluation.  Is it to improve services, generate new knowledge, comply with professional standards, and/or accountability?  The purpose will help select an evaluation framework that directly addresses the program’s needs to produce the expected outcomes for stakeholders.

Cool Tricks:

1. Use a Logic Model as a roadmap for understanding the program and the process for achieving the intended outcomes. The logic model is based on the Theory of Change that requires an evaluator to know the essentials of program theory and implementation to understand how a program works.
2. Link evaluation questions to the program outcomes as suggested by Rama Radhakrishna and Rhemilyn Relado in their article published in the Journal of Extension.

Lessons Learned:

My experience as an MSI Fellow has taught me about the ways and extent of rigor needed in program evaluation.  Conceptualization of rigor actually starts with asking the right questions.  Rigor is not just about using highly technical data collection and analyses methods but about selecting the best methods for the most critical questions regarding the process and outcome of the program to be evaluated.

The American Evaluation Association is AEA Minority Serving Institution (MSI) Fellowship Experience week. The contributions all this week to aea365 come from AEA’s MSI Fellows. For more information on the MSI fellowship, see this webpage: http://www.eval.org/p/cm/ld/fid=230 Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.