I am Vardhani Ratnala, a monitoring and evaluation professional. In this post, I would like to share my views on the importance of CONTEXT in evaluations.
Recently, at a high tea event, a friend pointed to a dress made from an Indian sari worn by an expat and suggested that we should get similar dresses stitched. In response, another friend pointed out that – “if an expat wore it, it might be considered fashionable, but if an Indian wore it, people would think that we were short of money and are recycling a sari into a dress”.
The conversation had me thinking on the importance of context. What might be considered positive in one context, can be considered average or negative in another context.
Lessons Learned: One can relate the importance of context to a number of evaluations. For example, in the context of a developed country, a disability programme providing a non-mechanical wheelchair might be considered an average intervention; but in a developing country context, where resources are limited, even provision of a tri-cycle, can be considered a life-altering intervention.
Prior to this event, I was discussing another evaluation with a friend. Our discussion centered on a programme offering legal assistance to trafficking victims to seek justice in a court of law. Very few victims had utilised the assistance, and only two of them had reached the verdict stage. Normally, the programme would have been considered a failure and its impact almost negligible. However, given the context in which the programme was operational, even the small numbers reached were remarkable. The programme was implemented in a region, where the police were non-cooperative, intimidation by traffickers was common, court cases dragged on for 10-15 years, and there was stigma associated with being identified as a trafficking victim. Under these circumstances, the programme was considered a success.
Hot Tips for context based evaluations: Apart from having a brief section on the context at the beginning of an evaluation report, it is essential to have “Context” as a specific evaluation criteria, so that the programme results can be viewed in the light of its social, cultural, political, legal or economic context, in order to determine its actual impact.
Since context is often subtle i.e. it is not always easy to articulate or observe, as there is a subtext involved, it is essential for evaluation teams to have a local evaluator on board, who can help understand the circumstances in which the programme was operational and thus, determine its impact.
Rad Resource: Check out this weblink for additional info: http://www.iisd.org/casl/caslguide/evalcontext.htm
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Great insight! In the international development space, there are the OECD DAC criteria to help standardize international evaluations for shared learning and comparability (in addition to other reasons). There is a criteria around relevance that typically relates to the needs and demands of the community for the intervention itself, to ensure the context of the intervention is not the limiting factor to the success of the program model (for example). I think as an evaluation community we can get better at addressing the relevance of context in our evaluation strategy, methodology as well as reporting – this would also raise the bar on ensuring the program designers are doing the same since contextual fit starts at the point of of putting the program together.
Ms. Ratnala, Your argument as to the importance of context in doing program evaluation is well made. Your example of the program for aiding trafficking victims is true. In the United States the program would be considered a failure, yet given the obstacles, cultural, political and administrative that the effort faced in the country it was implemented the few successes were significant. They point out that these programs can work, and alert the program leaders that changes may be necessary to make the program more effective. Failures can often teach as well as success. Therefore by recognizing the context the program was striving in the successes is appreciated and failures are better understood. Thus the whole effort is not just written off.
Another way of saying context could be “surrounding conditions” in which the program operates. I think those words would alert the cognizance of any reader of a program evaluation. The word “context” is used so much it has lost some of its semantic impact.