My name is Rob Fischer, and I am a faculty member at the Mandel School of Applied Social Sciences at Case Western Reserve University. I am pleased to contribute this blog post for AEA365’s focus on Evaluability Assessment. I teach evaluation to students in social work and nonprofit management and lead a number of studies of community-based initiatives.
When I was asked to contribute a post on this topic I admit that I was luke-warm to the idea. This is not because I do not buy into the value of EA. The idea of systematically assessing a program’s readiness for evaluation is inherently sensible and reflects a commitment to making the best use of finite evaluation resources. On the practical side, I meet with many programs and funders who would benefit from engaging in EA.
My experience is, though, that EA is one of the hardest sells in the evaluation business. Programs and funders often take it hard (and sometimes personal) when you suggest that they are not ready to embark on evaluation. In today’s outcome-focused environment, EA takes the endorsement from a clear-eyed funder before most programs will consider it. When programs seek an evaluation the recommendation is often felt as a setback. If I am ready to jump in my car to go on a trip and I am told I need to research destinations, examine route options and have my car checked out first, this may take the wind out of my sails. Are they smart things to do, without question, but it feels like so laborious. Such is the case with EA. But evaluation should not be about spontaneity, right?
Lesson Learned:
In my experience I have found that EA makes sense to many partners once they get a good dose of reality under their belt. EA can be framed as the smart way to inform any evaluation effort. It may be best understood as a process of ‘taking stock” – of such things as the program data that now exist, the degree of program clarity, and the status of program implementation. If a program requests an evaluation plan, all these things must be explored anyway. EA allows us to have a concerted effort applied to this undertaking. The downside is finding a partner who is willing to pay for EA. Ultimately, EA results in a plan to evaluate, not an evaluation report. I think the key is getting the partners to understand that they will get more value out of the subsequent evaluation effort if they invest in EA as a first step.
Rad Resource: I like this recent report from the UK on planning EAs, available open-access on their website.
The American Evaluation Association is celebrating CP TIG Week with our colleagues in the Community Psychology Topical Interest Group. The contributions all week come from CP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
I am interested in Rick Davies’ comment on “conflict of interest issues,” and whether it applies in my practice. I conduct a rapid EA as part of the design stage of large grant portfolio evaluations. There is still some pushback about spending as long on the design stage as I propose, but it helps to describe the rationale for each step in the assessment process without using the label, which I introduce once there is buy-in for all the steps! I frame the process as a thorough “reality check,” checking the logic of their TOC (again, often not using that label), often pointing out where which expected outcomes are too vague to be evaluable, or where there are huge logical leaps between activities and expected outcomes. The process ascertains which of the clients’ evaluation questions can be answered with the data available, whether the activities that are designed to produce a certain outcome have actually happened, and the possible methods and level of effort to answer some of the questions on medium- long-term outcomes. I offer to set up an M&E system so that the questions that cannot be answered at this point could be answered in the future.
Hi Bonnie
The reason for my wariness about a possible conflict of interest is that in the past some EAs have led to the (appopriate) abandonment of plans to carry out an evaluation, at least for some time.
Where the person doing the EA also has the contract to do the subsequent evaluation it may be much harder for them to turn around to their client and say that as a result of doing the EA they have concluded that the project is in fact unevaluable at this stage and the evaluation should not proceed. Doing so would mean they have just thrown away a contract.
My evidence base for this concern is very slim. One evaluation I was part of had a preliminary evaluability assessment built into it. Had that been done with more diligence there is a good chance the subsequent evaluation would not have gone ahead, something that subsequent events have since suggested would have been the best outcome.
regards, rick
Hi Rob
I agree with most of your analysis about the difficulty of selling a EA.
One part of the solution is to make sure that the costs of the EA are kept in proportion to the likely costs of any subsequent evaluation. If they are only 10% of the likely cost of a evaluation then it will not take a big improvement in the evaluation design to recoup these costs. Or, if an evaluation does not go ahead, the loss will be much less than having an evaluation go ahead and produce useless results. On the other hand if the cost of the EA is 50% of the likely cost of a evaluation then the impact will need to be much bigger and the savings from any avoided evaluation much less.
The other part of the solution is to ensure that the EA generates practical recommendations about what to do next, alongside any critique of the existing project (e.g. its Theory of Change, its data collection systems, its implementation progress). It is in this area that I have some difference of opinion. I am wary about the suggestion that an EA should “result in a plan to evaluate”. I think it would be preferable if an EA raised a series of issues, which are then incorporated in the ToR for an evaluation, and thus have to be addressed by those proposing to do the evaluation. This does not mean the EA cannot raise and review various evaluation method options, but I think it is risky if the EA ends up defining what other people will have to implement. The one exception might be where the EA is done by the same people who then do the evaluation. But this option does then raise potential conflict of interest issues.
regards, rick
Thanks, Rob – glad to see EA being discussed and Rick’s synthesis shared with AEA. I’m involved in developing a Guidance Note on EA for impact evaluations specifically, in which we have drawn on Rick’s work extensively. We are now testing these guidance notes in 4 cases, as part of process of engaging with bilateral agency projects in different countries around an intended impact evaluation. We will be sharing the Notes and our lessons soon. For those who are interested, you can get in touch with me iguijtatlearningbydesigndotorg.
Irene