I am Dawn Henderson, an Assistant Professor at Winston-Salem State University and former AEA GEDI (Graduate Education Diversity Initiative) intern. My experience as a GEDI afforded the opportunity to increase my knowledge in thinking about a program’s theory in evaluation. I used this in designing research and work with a school-based intervention. I want to share with you some tips I have used to help programs think about their particular theory and ways to evaluate and measure it. This is primarily targeted towards youth, so you can modify these tips for your specific needs.
Hot Tip: Conduct interviews with program staff and collect program materials. Interview key staff using questions like, What are unique characteristics of this program? How do these impact youth? When youth finish your program what do you want them to achieve or have? How would you measure that? These perspectives can assist you with understand factors that may potentially lead to outcomes. Review program materials such as curriculum, lesson plans, and activities, and organize them into common themes. For example, I read through the lesson plans of one program and identified managing conflict as a consistent theme and used this to help develop program outcomes.
I also used the findings to identify literature to support the program’s efforts. For example, I framed the activities of the program using research in positive youth development.
Hot Tip: Use visual aids. Use a visual aid to draw connections between program characteristics and outcomes. This can be created using basic shape features in Microsoft Word or SmartArt. I found that this visual aid helped the program think about how to achieve its objectives and communicate its model to external constituents.
Hot Tip: The “if” and “then” logic. It appears that everyone talks about using logic models in evaluation. Using the information you collected from program staff, materials, and visual aid, develop a series of “if” and “then” statements. For example, IF program X provides activities in conflict management and resolution THEN youth participants will improve their ability to manage and resolve conflict.
Hot Tip: The evaluation plan. Here is when you start gearing the program and yourself up for the evaluation. This can be similar to logic model development; I integrate each of the previous steps in this phase and include objectives, measures, participants, analysis, and outcomes. It not only helped me in thinking through the evaluation, but assisted the program in communicating their efforts to external constituents.
Rad Resource: Program Theory and Logic Models by Amhert H. Wilder Foundation
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Thank you for your helpful tips for developing a program’s theory for evaluation and measurement. I am a grad student at Queen’s University in Kingston, Ontario, and I am currently enrolled in a course called Program Inquiry and Evaluation. I am in the midst of designing a program evaluation, and as a novice evaluator, your post highlighted some great considerations for my evaluation design.
I chose to design an evaluation for the Kelso’s Choice conflict management skills program. Your examples of interview questions to ask the program staff were especially helpful for me, and I used them to guide my discussion of the program with a school guidance counsellor. I also found the ‘If…then…’ statements useful as I completed the logic model. I have survey questions designed for the teachers and some students, and I have created a formative assessment for the students. I am wondering if you have any specific tips you might offer for the analysis phase of an evaluation.
Thank you again for your useful post!
Susie using guiding questions with the organization you can co-facilitate the development of the program’s theory of change.
Thanks for this post. Useful.
But a question: Seems that you are presuming there is a theory of change for the program you are evaluating. What if there isn’t one or there is one but no one involved with the program knows what it is (in the case of a program that’s been around for a while and none of the staff were there at the beginning)? I’m aware of numerous programs like this.
Thanks for your help!