My name is Jane Davidson and I run an evaluation consulting business called Real Evaluation Ltd. In my work, I advise and support organizations on strategic evaluation; provide evaluation capacity building and professional development; develop tools and templates to help organizations conduct, interpret, and use evaluations themselves; and conduct independent and collaborative evaluations and meta-evaluations.

Over several years’ working with clients and reviewing (at clients’ request) disappointing evaluation reports, I have noticed several critically important elements that make or break evaluation work but are often missing from evaluators’ methodological toolkits.

Hot tip: Clients find it incredibly frustrating to wade through an evaluation report full of evidence and still be none the wiser at the end whether the documented outcomes (let alone the entire program/policy/etc) are any good or not. A key part of an evaluator’s work is to say clearly and explicitly how practically, educationally, socially, or economically (not just statistically) significant outcomes are (severally, and as a set). This is what makes evaluation ‘e-VALU-ation’!

Hot tip: A useful tool for generating real evaluative conclusions is an evaluative rubric. This is a table describing what different levels of performance, value, or effectiveness ‘look like’ in terms of the mix of evidence on each criterion. Grading rubrics have been used for many years in student assessment. Evaluative rubrics make transparent how quality and value are defined and applied. I sometimes refer to rubrics as the antidote to both ‘Rorschach inkblot’ (“You work it out”) and ‘divine judgment’ (“I looked upon it and saw that it was good”)-type evaluations.

Hot tip: Collaborative development of rubrics is a great way to get stakeholders thinking about how ‘quality’ and ‘value’ should be defined for the work they do. It helps build the evaluative thinking needed to generate, understand, accept, and use evaluation findings.

Rad resources:

• Evaluation Methodology Basics: The nuts and bolts of sound evaluation by E. Jane Davidson (2005)
• Improving evaluation questions and answers: Getting actionable answers for real-world decision makers (AEA e-Library’s most viewed and downloaded item)
• Example rubrics in Nunns, Roorda, et al’s (2010) Evaluation of the Recognised Seasonal Employer Policy
• Example rubric (referred to as a ‘global assessment scale’) developed for the evaluation of the Corangamite Salinity Program (case study #10 in Jessica Dart et al’s 1998 Review of Evaluation in Agricultural Extension, pp. 62-63 – a publication from the Rural Industries Research and Development Corporation)
• AEA conference professional development workshop about how to use rubrics (and other evaluation nuts and bolts) to do actionable evaluations (November 10, 2010)
• Strategic evaluation of the workplace assessment program, a relevant recent chapter from Jane in the Handbook of Workplace Assessment (2010)

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org. Want to learn more from Jane? She’ll be presenting as part of the Evaluation 2010 Conference Program, November 10-13 in San Antonio.

Rubric, Toolkit

Hi, my name is Christopher Moore.  I am a doctoral student in Quantitative Methods in Education at the University of Minnesota and a Quantitative Analyst at the Minnesota Department of Education.  My interests include preventing educational and health disparities, latent variable models, spatial statistical methods, and causal theory and inference.

Hot Tip: So you’re conducting a theory-driven program evaluation?  You’ve developed a solid logic model, you’ve collected relevant quantitative data, and now you’re interested in estimating the degree to which the program has been effective?  Structural equation modeling is a statistical approach that is well-suited for estimating relationships specified by a logic model.

As described by Paul Mattessich in The Manager’s Guide to Program Evaluation, logic models feature program elements and paths from causal elements to outcomes.  Elements in the middle represent both causes and outcomes, mediating the influence of inputs on longer-term outcomes.  Theory-driven evaluators like to pull mediators out of the “black box.”

Figure 1. Elements of a logic model

In the analysis phase of a theory-driven evaluation, structural equation modeling can simultaneously operationalize elements as latent factors and estimate multiple causal paths.  It does so by modeling the observed covariance matrix.  If the data contain dichotomous or ordinal dependent variables, then a polychoric correlation matrix should be modeled.  A sequential strategy (e.g., scaling followed by regression analysis for each dependent variable) requires more steps and can underestimate causal paths by not accounting for measurement error.

A logic model can be adapted into a structural equation model path diagram (see Figure 2).  Observed variables are represented by rectangles, and latent variables are represented by ellipses.  For simplicity, the example below features no error terms and only one input, activity, output, and outcome.  The outcomes are treated as latent variables reflected by repeatedly observed indicators (e.g., survey questions).  The intercept and slope capture initial status and change over time, respectively.

Figure 2. A partial mediation growth model adapted from a logic model

Moving to a real-world scenario in which structural equation modeling could be applied, Kathryn Tout and colleagues at Child Trends have identified a need for theory-driven evaluations of child care Quality Rating Systems (QRS).  QRS represent a relatively new approach to helping parents choose high quality child care, which is believed to promote child development.  Using Tout and colleagues’ article as a guide, I developed a path diagram that could be estimated with data being collected by QRS evaluators.  The actual path diagram would have more inputs, outputs, and item scores.

Figure 3. A path diagram for evaluating a child care Quality Rating System

Structural equation modeling requires familiarity with matrix algebra and formal training in latent variable models and related software.  Melanie Wall, David Garson, and Alan Reifman have created helpful course web pages.  Amos is a popular add-on to SPSS that can specify structural equation models by drawing path diagrams.  Mplus is another popular program and my favorite because it can handle multilevel, categorical data sampled in a complex manner (i.e., with unequal probabilities of selection), although it does not produce path diagrams.  The sem package in R is free and another favorite of mine.  When using Mplus or the sem package, Graphviz can be used to create path diagrams, as I did above.

I hope this “tip” has encouraged you to at least consider structural equation modeling during the data collection and analysis phases of a theory-driven evaluation.  Even though evaluators skillfully develop theories of change that recognize multiple causes and outcomes inside the “black box,” a search of evaluation publications suggests that structural equation modeling could be utilized more fully.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

logic models, SEM, structural equation model, theory-driven

Hello, I am Glenn O’Neil and specialize in evaluating communication programs and campaigns with my own company, Owl RE. My post today is about how to use the theory of change in evaluating communication programs.

Hot tip: there is nothing so practical as a theory of change! The theory of change maps out from activities to impact how the communications action would bring about change, often in a flow-chart like diagram. Here is a simplified example:

This should be done when designing a communications action but in my experience it is rarely the case. So you can reconstruct the theory of change at the start of the evaluation – what activities were undertaken? What was the desired short and long term effects – for example, raising awareness amongst whom? Getting people to act, but on what? Mobilizing publics – but what for? This helps clarify what you are then going to measure and how to go about it.

Rad resource: For more examples of how the theory of change is used in campaign evaluation for non-profits, check out this excellent paper from Julia Coffman of Harvard University: “Lessons in evaluating communications campaigns: Five case studies. Harvard Family Research Project, 2002 (pdf)”.

For those that would like a broad overview of ‘how to’ evaluate communication programs and projects, check out my presentation slides for a “One day training workshop for communication professionals on evaluating communication programmes, products and campaigns”.

communications, Models, Theory of Change