Greetings from Ian Patrick and Anne Markiewicz, in Melbourne, Australia – evaluators active in evaluation design, implementation and training for a range of domestic and international clients. We’ve been reflecting on a tortured area of evaluation practice – that being expectations frequently placed on evaluators to identify the IMPACT of a program.
Every evaluator breathes a sigh of relief when their clients or stakeholders are knowledgeable about evaluation and hold reasonable expectations about what it can and can’t do. But how many evaluators instead have felt the heavy weight of expectations to establish high level results demonstrating a program has make a big difference to a region, country or the world! Or in a related scenario, an eagerness to establish longer term results from a program which has only been operating for a limited duration! Other unrealistic expectations can include adopting a program-centric focus which sees all results as attributable to the program, minimizing the contribution of stakeholders and partners to change. Or in another context, adopting a limited lens on the perceived value of different types of results.
Such situations call for cool-headedness and a calm educative approach from the evaluator. Where possible, the evaluator has much to gain from open discussion and exchange of views, tempering unrealistic aspirations and negotiating realistic expectations from an evaluation. Here are some of the strategies that we have found productive in such contexts:
HOT TIPS:
Reflect on Impact: As an upfront strategy, become clear with clients/stakeholders about what is meant by ‘impact’. Be aware, that the term is used loosely, and lazily, often to support sweeping expectations. Introduce other helpful terminology to identify and demarcate different categories of results such as intermediate outcomes. A sense of realism in discussions may well clarify that these types of results can be realistically identified within the program time frame. Intermediate results, once identified and understood are often highly valued, and stand in contrast to more elusive, longer term impact.
Decompress Time: Proactively address a tendency for time factors associated with a program’s results to become compressed. A fixation on end states can obscure the important intermediary stages through which change evolves. Utilisation of program theory and program logic approaches can provide a means to identify expected changes over realistic time frames.
Remember Others: Resist a tendency for change to be unilaterally attributed to a program. Recognise and focus on the contribution made by related stakeholders/partners to change.
Adopt Pluralist Approaches: promote application of various perspectives and ways of identifying and measuring change rather than using a single method. Use of mixed methods approaches will promote a more subtle and nuanced view of change, particularly how it manifests and is experienced during a program’s life cycle.
This contribution is from the aea365 Tip-a-Day Alerts, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org. Want to learn more from Ian and Anne? They’ll be presenting as part of the Evaluation 2014 Conference Program, October 15-18 in Denver, Colorado.