Greetings from the Center for Research Evaluation! I am Sarah Mason—an Australian evaluator working in Mississippi—here to talk about contextually responsive evaluation.
Like many evaluators, I’ve had regular conversations about the idea that context matters in evaluation.
Context matters in our line of work because there’s no universal ‘right way’ to do an evaluation. Instead, evaluation contexts necessarily affect the design, implementation and effectiveness of our evaluation practice.
In 2018, the AEA formally recognized this idea as a key evaluation competency, dedicating an entire competency domain to context (the Context Domain) and charging evaluators with responsibility for understanding and responding to the contexts in which they work.
What exactly is context?
But what, exactly, do people mean when they talk about context in evaluation? And what, specifically, should an evaluator look for if we want to respond to the uniqueness of our evaluation context?
Hoping to gain some clarity on these questions, I took to the (metaphorical) streets, interviewing 12 AEA award winners and surveying more than 400 evaluators from Australia, New Zealand and the United States with the goal of creating a checklist that might be useful in guiding contextually responsive practice.
Lessons Learned: So what do evaluators mean when they talk about context?
At least 70 different things! Evaluators from these three countries identified 70 pieces of contextual information that evaluators could (and should) pay attention to if we want to understand—and respond to—our evaluation contexts.
These fell under seven categories, including:
- The program we’re evaluating (e.g. What does the program do?)
- The organization delivering the program (e.g. Who is in charge?)
- The broader systems and structures within which the program operates (e.g. Who else does the program depend on to get things done?)
- The evaluation itself (e.g. What decisions will it inform? How much is at stake?)
- The evaluation team (What are our biases? How well do our capabilities fit the needs of this specific evaluation?)
Cutting across all these categories, we also need to pay attention to:
- The personal and interpersonal relationships embedded in our evaluation, and
- The political nature of our evaluation (e.g. Who stands to win (or lose) from positive (or negative) findings?)
The Framework for Situation Awareness in Program Evaluation summarizes these 70 contextual factors and describes a more detailed framework for thinking about what it means to conduct contextually responsive evaluation.
The American Evaluation Association is celebrating Research on Evaluation (RoE) TIG Week with our colleagues in the RoE Topical Interest Group. The contributions all this week to aea365 come from our RoE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org.