CAP Week: Jack Mills on Evaluation Theory

My name is Jack Mills; I’m a full-time independent evaluator with projects in K-12 and higher education. I took my first course in program evaluation in 1976. After a career in healthcare administration, I started work as a full-time evaluator in 2001. The field had expanded tremendously in those 25 years. As a time traveler, the biggest change I noticed was the bewildering plethora of writing on theory in evaluation. Surely this must be as daunting for students and newcomers to the field as it was for me.

Rad Resource: My rad resource is like the sign on the wall at an art museum exhibit—that little bit of explanation that puts the works of art into a context, taking away some of the initial confusion about what it all means. Stewart Donaldson and Mark Lipsey’s 2006 article explains that there are three essential types of theory in evaluation: 1) the theory of what makes for a good evaluation; 2) the program theory that ties together assumptions that program operators make about their clients, program interventions and the desired outcomes; and 3) social science theory that attempts to go beyond time and place in order to explain why people act or think in certain ways.

As an example, we used theory to evaluate a training program designed to prepare ethnically diverse undergraduates for advanced careers in science. Beyond coming up with a body count of how many students advanced to graduate school, we wanted to see if the program had engendered a climate that might have impacted their plans. In this case, the program theory is that students need a combination of mentoring, research experience, and support to be prepared to move to the next level. The social science view is that students also need to develop a sense of self-efficacy and the expectation that advanced training will lead to worthwhile outcomes, such as the opportunity to use one’s research to help others. If the social science theory has merit, a training program designed to maximize self-efficacy and outcome expectations would be more effective than one that only places students in labs and assigns them mentors. An astute program manager might look at the literature on the sources of self-efficacy and engineer the program to reinforce opportunities that engender it.

This aea365 contribution is part of College Access Programs week sponsored by AEA’s College Access Programs Topical Interest Group. Be sure to subscribe to AEA’s Headlines and Resources weekly update in order to tap into great CAP resources! And, if you want to learn more from Jack, check out the CAP Sponsored Sessions on the program for Evaluation 2010, November 10-13 in San Antonio.

5 thoughts on “CAP Week: Jack Mills on Evaluation Theory”

  1. This was a very interesting and engaging article, especially for someone who has no experience in this area. I didn’t even know that a field called program evaluation even existed until I needed to take this required course. There is a social program where I live called the Portage Learning and Literacy Centre (PLLC). They have a plethora of sub-programs available such as adult education (they work on a high school diploma), employment assistance, work experience program and newcomer settlement services (for immigrants new to our area) among other programs. I was interested about the program for ethically diverse undergrad students. While the target demographic (undergrad students v. new immigrants) may be different, both the PLLC program and you program do share the ethnically diverse nature of the program and helping those people navigate unfamiliar territory.

    You mentioned that the field of evaluation has expanded greatly. What in your opinion can you attribute this explosion?

  2. Pingback: Innovation Integration & Self-Efficacy (#7) – Dimitra’s Reflections

  3. Pingback: Module 1: Part C – Response to J. Mills – Professional Master of Education Journey

  4. Pingback: Module 1 : Introduction to Program Inquiry A. Foundational Knowledge – Alishia Ellis

  5. Jack, this is an excellent post addressing a critically important topic in a novel way. As a professor of program evaluation, I very much appreciate your analogy of program theory to the explanation on the wall in an art museum. That is an excellent and creative metaphor that I will certainly borrow when I again introduce the concept of program theory to a group of novice evaluators in my fall course! Thank you!

Leave a Reply to Sheila Cancel Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.