My name is Élyse McCall-Thomas, PhD Candidate at the University of Ottawa. In this post, I reflect on how working outside traditional timelines and closely with programs can facilitate the use of evaluation findings.
In the Canadian federal government, policy dictates that evaluations of grant and contributions programs must occur every five years and be conducted in a neutral manner. While working at the Natural Sciences and Engineering Research Council, a federal agency that funds research and innovation, the evaluation division was approached by Ed Irving and Jeffrey Nerenberg, the directors of the newly implemented Alliance Grants, who wanted to understand why participation was lower than anticipated. They had theories but wanted evidence to confirm their thinking and support potential program changes. Although the formal, policy-required evaluation was still two years away, they needed answers now, and it was our responsibility to help them.
When Timing Is Everything
To meet the program’s needs, my director, Susan Morris, suggested a rapid program review to generate quality evidence quickly for learning and decision-making. The review would be narrowly focused and completed in six months.
Rather than producing a formal written report with recommendations, we summarised our findings and opportunities for improvement in a presentation deck. This made the information easier to digest and faster to share. The program could also repurpose the deck for senior management, and other interested parties.
Hot Tip
- If a rapid review is conducted outside the formal evaluation cycle, plan from the outset for how the findings will feed into the eventual evaluation. This keeps the work rigorous and defensible.
A More Participatory Approach for Useful Findings
My colleagues, Kelly-Lyn Christie and Sara Grondin, and I worked closely with the program to identify what they needed to know, maintaining ongoing dialogue about emerging findings and ideas for change.
We found opportunities to join consultations the program was already planning with the research community. This was a novel approach for us as government evaluations are often conducted independently to meet policy expectations around neutrality. We supplemented the consultations with independent data collection, keeping the work efficient without sacrificing rigour.
Because of their involvement in the review, the program was able to make quick use of the findings, redesigning elements of the Alliance Grants to increase uptake. They called this the Alliance Refresh. In my eighteen years as an evaluator, this was the first time I witnessed such timely and concrete use of evaluation findings to inform program decisions.
Hot Tip:
- Before designing your data collection, ask the program what engagement activities they already have planned and consider joining those alongside your own independent data collection. This saves time, reduces burden on interested parties, and builds trust.
Lessons Learned
If the organization has an evaluation policy or specific expectations for how evaluations are to be conducted, it helps to know those up front and identify potential “gray zones.” Flexibility is only possible if you understand what is expected and how you can meet those expectations without sacrificing the utility of evaluation.
Rad Resources
- Interested in hearing more? Ed, Jeff and I discuss this review and how the findings were used in Managing for Results in Science, Research and Innovation.
- Jill Chouinard’s article, The Case for Participatory Evaluation in an Era of Accountability, highlights the tension between participatory approaches and the accountability-driven, neutrality-focused culture of public sector evaluation, and makes the case why the former approach still has a place in this context.
Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please do so using the form on our website. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.