Greetings from the Center for Research Evaluation! I am Sarah Mason—an Australian evaluator working in Mississippi—here to talk about contextually responsive evaluation.
Like many evaluators, I’ve had regular conversations about the idea that context matters in evaluation.
Context matters
Context matters in our line of work because there’s no universal ‘right way’ to do an evaluation. Instead, evaluation contexts necessarily affect the design, implementation and effectiveness of our evaluation practice.
In 2018, the AEA formally recognized this idea as a key evaluation competency, dedicating an entire competency domain to context (the Context Domain) and charging evaluators with responsibility for understanding and responding to the contexts in which they work.
What exactly is context?
But what, exactly, do people mean when they talk about context in evaluation? And what, specifically, should an evaluator look for if we want to respond to the uniqueness of our evaluation context?
Hoping to gain some clarity on these questions, I took to the (metaphorical) streets, interviewing 12 AEA award winners and surveying more than 400 evaluators from Australia, New Zealand and the United States with the goal of creating a checklist that might be useful in guiding contextually responsive practice.
Lessons Learned: So what do evaluators mean when they talk about context?
At least 70 different things! Evaluators from these three countries identified 70 pieces of contextual information that evaluators could (and should) pay attention to if we want to understand—and respond to—our evaluation contexts.
These fell under seven categories, including:
- The program we’re evaluating (e.g. What does the program do?)
- The organization delivering the program (e.g. Who is in charge?)
- The broader systems and structures within which the program operates (e.g. Who else does the program depend on to get things done?)
- The evaluation itself (e.g. What decisions will it inform? How much is at stake?)
- The evaluation team (What are our biases? How well do our capabilities fit the needs of this specific evaluation?)
Cutting across all these categories, we also need to pay attention to:
- The personal and interpersonal relationships embedded in our evaluation, and
- The political nature of our evaluation (e.g. Who stands to win (or lose) from positive (or negative) findings?)
Rad Resource:
The Framework for Situation Awareness in Program Evaluation summarizes these 70 contextual factors and describes a more detailed framework for thinking about what it means to conduct contextually responsive evaluation.
The American Evaluation Association is celebrating Research on Evaluation (RoE) TIG Week with our colleagues in the RoE Topical Interest Group. The contributions all this week to aea365 come from our RoE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org.
Hello Sarah,
This was interesting information to know. Awesome job doing this research and getting to the root of what evaluators perceive as context! This can definitely bring clarity when beginning to prepare for a new program.
Hello Sarah!
Thank you for sharing this post on the importance of contextual understanding when carrying out an evaluation. The AEA365 Blog has been a key resource throughout the course I am working through for my Professional Master of Education program. You very clearly summarized the various factors that can affect the context in which we are conducting an evaluation.
In Practical Program Evaluation, the author states, Evaluation becomes productive only when we go beyond methodology to ply the waters with theoretical and also contextual knowledge” (Huey-Tsyh 3). You and Huey-Tsyh both emphasize the multi-faceted nature of evaluation, as context extends beyond just the program, and includes that of the organization, broader systems, the evaluation, the evaluation team, and more. Understanding the context of a program sets evaluators on the critical path for obtaining the best results, and for making the best use of findings.
Thank you for sharing an excellent resource!
Huey-Tsyh, C. (2005). Practical program evaluation: Assessing and improving planning, implementation, and effectiveness (pp.ix-13). Thousand Oaks, CA: Sage Publications.
Hello Sarah,
I am a student from Ontario, Canada, currently enrolled in a Masters of Education Program through Queen’s University. One of our courses specifically deals with Program Evaluation and the AEA 365 Blog is one which is used to aid in our learning. I wanted to thank you for your post which deals with an issue important to Program Evaluation. You have succinctly explained the reason why context plays an important role in program evaluation and identified many contributing reasons. I am reminded of an article by Carol Weiss (1998), “Have We Learned Anything New about the use of Evaluation?” in which she states, “But we have come to a growing realization of how complicated the phenomenon of use is and how different situations and agencies can be from another.” Weiss’s comment stood out for me when reading your blog as I came to realize how social constructs have shaped program evaluation throughout history. Therefore, the context becomes important for evaluators to understand when evaluating the importance of a program or policy. I especially enjoyed reading through your Rad Resource and appreciated how you have isolated pertinent areas and questions for an evaluator to understand when completing an evaluation. Thank you for this great post and resource!
Weiss, C. H. (1998). Have we learned anything new about the use of evaluation?
American Journal of Evaluation, 19, 21-33.
Thank you Sarah, that’s terrific work, sound food for thought.
I look forward to working through the ‘Framework for Situation Awareness in Program Evaluation’ and seeing how I can use it (with the source suitably acknowledged, of course!) in my own practice and writing.