Hello! I am Terence Fitzgerald, Senior Director of Program Design & Evaluation at International Justice Mission (IJM), a nonprofit organization that works to protect people from violent crime. I lead a team of staff that provides leadership and technical support to IJM on program design, monitoring, evaluation, and research. My team’s remit is to “bring evidence to bear for the mission” so that leaders at executive, portfolio, and project levels can make informed decisions; and so that IJM can realize its vision and fulfil its mission. I want to share lessons that my team has learned from working on program design from inside a mission-driven organization.
Lessons Learned:
- Demystify program design: Before we engage with a team on program design, we ask for and listen to their views and experiences of it; and, where necessary, we explain concepts, processes, definitions, benefits, and challenges. We meet the teams where they are. I have found that a one-hour session on designing a personal program around “being more healthy” can generate lots of input and allow for discussion of many core design issues – different types of impact; relationships between resources, activities, and results; milestones and indicators; and assumptions and risks. Even with inexperienced staff, it can be quite easy to convey key concepts, get enthusiastic buy-in, and then build a more solid design.
- Progressively elaborate the design: As we work with teams on a design, my team creates draft logic models and other design artifacts based on our understanding of what they have told us. We give those back to the team to review our work, correct any erroneous content, note any unresolved issues, and to secure buy-in for further elaboration. We raise concerns where we see threats to the design’s plausibility or feasibility, and we help teams to understand the causes of our concerns. We raise unresolved concerns to decision-makers.
- Facilitate change to designs: Teams can tend to revere and thus be reluctant to change the designs into which they have put time and energy. My team reinforces that designs are meant to change, based on the team’s experiences and environmental changes. We explain and, where necessary, guide teams through IJM’s program change management process.
Rad Resources:
- The OECD DAC criteria for evaluating development assistance are very useful to bring evaluation into the program design process. My team uses the criteria to inform design discussions that often lead to clarifications on program priorities, inclusions and exclusions, and other design elements.
- Rachel Kleinfeld’s Improving Development Aid Design and Evaluation: Plan for Sailboats, not Trains is an excellent resource for those designing complex programs centered on political change. It highlights several design errors that my team has seen and that we seek to avoid.
The American Evaluation Association is celebrating Program Design TIG Week with our colleagues in the Program Design Topical Interest Group. The contributions all this week to aea365 come from our Program Design TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hello Terence,
I am a student enrolled in Queens Universities Professional Masters in Education Program. I am currently studying Program Evaluation, and my final project will be a Program Evaluation Design. The program I have chosen to evaluate is Sunrise of Life (sunriseoflife.org), whose primary aim is to help the street children of Arusha, Tanzania. My program inquiry & evaluation website provides more information about SOL, the program theory, logic model and the method in which I could collect data for this evaluation: janinevanmaren.wordpress.com.
However, working with a non-profit in a vastly different culture from my own has had its own set of challenges as well. For my evaluation I have spoken with several of the stakeholders of the organization, but after reading your article I was reminded of the importance of having this be a truly participatory process. If it were financially feasible it would be best to meet with the team as a whole and really discuss what ‘they’ need, rather than what I think they need. I appreciated the focus on creating a logic model that then gets reviewed, corrected and changed based on what the different groups have reported.
The DAC criteria for evaluating development assistance (relevance, effectiveness, efficiency, impact, and sustainability) have given me more questions, rather than answering the many I already have. These questions, I believe, are what will really add value to the research being done. Working with money provided solely by funding partners in the western world, attempting to help street youth in Tanzania as best we can, means that each of these five criteria need to be carefully pondered over. The one thing that stood out to me in this list was the criteria about sustainability: It is here that I really feel the organization is lacking, but I am also unsure about how to change this from being a fully funded organization being run mostly by western partners, and into one that is self-sustaining both in terms of running it, but also financially.
The sailboat analogy in the Carnegie Endowment for International Peace is one that I have shared with the team at SOL: we indeed have weeks where we say “look at how well this youth is doing”, and “we received new funding to improve the dormitories”… but then conversely we have weeks of only setbacks. The path of a non-profit is far from smooth and straight, but as long as the general direction we are headed is towards our end mission statement, then I am happy to be on that sailboat.
Thank you for your thought-provoking article!
Janine Van Maren
15jnvm@queensu.ca