Hi, I am Sara Vaca (Independent Evaluator and Saturday’s contributor to this blog).
One concern evaluators and commissioners share is how to operationalize the evaluation questions. That is: how to articulate that the link between the evaluation questions (what we want to know or confirm) and the data collection (what we are actually asking and what answers/information we are getting) and analysis processes (how we make sense of it all), that will lead to the evaluation results (findings, conclusions and recommendations).
Rad Resource: Some evaluators and commissioners use an Evaluation Matrix.
I initially thought that everybody in evaluation used these but I realized not long ago that it is an unfamiliar tool for some in our community, so I decided to write this post.
An evaluation matrix is a table where each evaluation question is linked to the method/s that will provide information to answer them. As illustrated in these short guidelines, it is the evaluation plan.
Hot Tip: An evaluation matrix often includes a column for indicators or setting the criteria to judge the data collected (something I find extremely difficult to do before the data collection and hard to rely on it and use later, so I hardly ever do it).
Here are several types or models I’ve come across:
|Evaluation Question||Sub-Question||Indicator /Criterion||Data Source||Data Collection Method||Sampling Plan|
|Hyphothesis 1||Indicators||Information sources||Data collection tools|
Option C – the one I use (I use a column per method so I can filter them later):
Lesson Learned: When someone first came up with the idea of using an evaluation matrix, they must have definitely meant a development in articulating the evaluation logic, however… the problem I encounter using them is that most of the questions are using the same methods, so their practical utility decreases.
Hot Tip: Trying to make them more useful for my practice, I have incorporated another feature – a new set of columns with the stakeholders or sources that will answer them. Like this:
For me it is useful to filter by stakeholder to know what to ask each group and to elaborate the evaluation tools (questionnaires, and interviews guidelines). Still, there seem to be too many questions that seem to be using the same methods… And I always wonder: is this OK?
So my question is: How can we improve the Evaluation Matrixes? Tips, resources or discussion around them are most welcome!
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.