AEA365 | A Tip-a-Day by and for Evaluators

TAG | process evaluation

Greetings from Colorado – Home of the Southern Rocky Mountains and the edge of the Great Plains. My name is Helen Holmquist-Johnson and I am the Assistant Director of the Social Work Research Center at Colorado State University in Fort Collins, Colorado. Colorado is not only geographically diverse, but incredibility diverse in terms of the demographic characteristics of the communities and the individuals who live here. Some of my recent work focuses on evaluating evidence-based programs that promote strengthening families and keeping children safe and healthy.

In Colorado, the Division of Child Welfare is a state supervised, county administered system. In fact, Colorado is one of only nine states in the U.S. where the administrative structure of Child Welfare can be described this way. Each county, while held to the same State and Federal requirements, can individually decide how to operate and deliver child welfare services to families. This arrangement increases the level of autonomy the counties have on everything from which programs they implement, to utilizing different models of leadership and supervision. In a state with 64 counties (I know some of you are already ahead of me here), this structure introduces some unique evaluation challenges.

This is where process evaluation becomes useful and important. What might be missed or overlooked in an outcomes evaluation can be captured by asking process evaluation questions. In general, these questions focus on who the program reaches and whether or not the program was carried out as planned. Because we are evaluating evidence-based models, our interest shifts somewhat from asking does the program work, to asking how, why and in what context or conditions does the program work? As you can see, these are the questions which are important to ask when working in 64 different counties throughout the state. Answers to these questions will assist county administrators and other stakeholders in making policy and practice decisions which consider the contextual factors unique to their community and families.

Rad Resource: For specifics about how to design and conduct a process evaluation, read Steckler, A., and Linnan, L. (Eds.), Process Evaluation in Public Health Interventions and Research.

Hot Tip for Denver: If you have children and family coming with you to Evaluation 2014 in Denver you might want to check out The Children’s Museum of Denver and The Butterfly Pavilion.

We’re looking forward to October and the Evaluation 2014 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to


I’m Catherine (Brehm) Rain of Rain and Brehm Consulting Group, Inc., an independent research and evaluation firm in Rockledge, Florida. I specialize in Process Evaluation, which answers the questions Who, What, When, Where and How in support of the Outcome Evaluation. Field evaluations occur in chaotic environments where change is a constant. Documenting and managing change using process methods help inform and explain outcomes.

Lesson Learned: If you don’t know what or how events influenced a program, chances are you won’t be able to explain the reasons for its success or failure.

Lesson Learned: I’m a technology fan, but I’m also pretty old-school. Like Caine in the legendary TV show Kung Fu, I frequently conjure up the process evaluation ‘masters’ of the 1980s and ‘90s to strengthen the foundation of my practice and to regenerate those early ‘Grasshopper’ moments of my career.

Old-school? Or enticingly relevant? You decide, Grasshopper! I share a few with you.

Hot Tip:  Process evaluation ensures you answer questions of fidelity (to the grant, program and evaluation plan): did you do what you set out to with respect to needs, population, setting, intervention and delivery? When these questions are answered, a feedback loop is established so that necessary modifications to the program or the evaluation can be made along the way.

Rad Resource: Workbook for Designing a Process Evaluation, produced by the State of Georgia, contains hands-on tools and walk-through mechanics for creating a process evaluation. The strategies incorporate the research of several early masters, including three I routinely follow:  Freeman, Hawkins and Lipsey.

Hot Tip: Life is a journey—and so is a long-term evaluation. Stuff happens. However, it is often in the chaotic that we find the nugget of truth, the unknown need, or a new direction to better serve constituents. A well-documented process evaluation assists programs to ‘turn on a dime’, adapt to changing environments and issues, and maximize outcome potential.

Rad ResourcePrinciples and Tools for Evaluating Community-Based Prevention and Health Promotion Programs by Robert Goodman includes content on the FORECAST Model designed by two of my favorites (Goodman & Wandersman), which enables users to plot anticipated activities against resultant deviations or modifications in program and evaluation.

Hot Tip:  If you short shrift process evaluation, you may end up with Type III error primarily because the program you evaluated is not the program you thought you evaluated!

Rad Resource: Process Evaluation for Public Health Research and Evaluations: An Overview by Linnan and Steckler discusses Type III error avoidance as a function of process evaluation. As well, the authors discuss the historical evolution of process evaluation by several masters including but not limited to Cook, Glanz and Pirie.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·


To top