Hello! We are Monica Hargraves and Miranda Fang, from the Cornell Office for Research on Evaluation. We presented together at Eval2012 would like to share some practical tips on literature searches in the context of evaluation.
Program managers often face an expectation worthy of Hercules: to provide strong research-quality evidence that their program is effective in producing valuable outcomes. This is daunting, particularly if the valued outcomes only emerge over a long time horizon, the program is new or small, or the appropriate evaluation is way beyond the capacity of the program. The question is, what can bridge the gap between what’s feasible for the program and what’s needed in terms of evidence?
Hot Tip: Strategic literature searches can help. And visual program logic models provide an ideal framework for organizing the search process.
Quoting our colleagues Jennifer Urban and William Trochim in their AJE 2009 paper on the Golden Spike,
“The golden spike is literally a place that can be drawn on the visual causal map … where the evaluation results and the research evidence meet.”
We use pathway models, which build on a columnar logic model and tell the logical story of the program by specifying the connections between the activities and the short-term outcome(s) they each contribute to, and the subsequent short- or mid-term outcome(s) that those lead to, and so on. What emerges is a visual program theory with links all the way through to the program’s anticipated long-term outcomes.
The visual model organizes and makes succinct the key elements of the program theory. It helps an evaluator to zero in on the particular outcomes and causal links that are needed in order to build credible evidence beyond the scope of their current evaluation.
Here’s an example, from a Cornell Cooperative Extension program on energy conservation in a youth summer camp. Suppose the program needs to report to a key funder whose interest is in youth careers in the environmental sector. If the program evaluation demonstrates that the program is successful in building a positive attitude towards green energy careers, then a literature search can focus on evidence for the link (where the red star is) between that mid-term outcome and the long-term outcome of an increase in youth entering the green workforce.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
This is a neat idea for constructing a visual of how a program works, thanks. An idea for applying this to evaluating different types of programs is that, depending on the nature of program, the models could be expanded to include non-linear arrows going in the other direction. Also, some programs and initiatives might need multiple models that are tied together, to show multiple levels of program activities and outcomes.
A scan of the literature can be helpful not only for examining the evidence for the links after the model is constructed, but also in the initial step of clarifying the program theory and identifying what elements might be relevant to include in the model.