My name is Cassie Bowman. As the coordinator of NASA’s Mars student intern program, I’ve looked for ways to continuously improve the experience for the student, teacher, and scientist participants. My tip is that empowerment evaluation, used in a modified format, can be an excellent way to make improvements to a program that has constantly changing membership.
Though empowerment evaluation is most commonly used to build capacity and foster improvement among stakeholders and staff in a group or organization, NASA’s Mars Public Engagement Program has found it can be used very successfully to evaluate and improve a program with changing group membership. We have used empowerment evaluation in our Mars student intern program with each year’s group of students and teachers—the first year under the direct guidance of Dr. David Fetterman and subsequent years on our own (I guess that’s the capacity-building part!).
The standard model of empowerment evaluation involves three main steps (Fetterman, 2001):
- Step 1: “Developing a mission, vision, or unifying purpose”
- Step 2: “Taking stock or determining where the program stands, including strengths and weaknesses”
- Step 3: “Planning for the future by establishing goals and helping participants determine their own strategies to accomplish program goals and objectives.”
Though it sounds highly conceptual, these steps are carried out in very concrete ways. For Step 1, if the group engaged in the evaluation is meeting face-to-face, they might go about drafting (or editing) a mission/vision/unifying purpose together in real-time. In our case, because our student/teacher groups are distributed across the country, we have people submit their ideas electronically, then share drafts back and forth before coming to consensus via teleconference. For Step 2, after brainstorming a long list of program aspects/elements, those meeting in person are given 10 stickers that they use to “vote” on aspects of their program that are the most important by placing the stickers next to aspects on the list that they value. Once the list is whittled down to a reasonable number through this voting process, participants rate each aspect (usually on a scale from 1-10) and then discuss the ratings. In our case, online voting or survey software is used for both the voting and rating, and discussions are held via teleconference. Finally, as with the previous steps, Step 3, planning for the future, can be held in person or via distance technologies.
We have found empowerment evaluation to be a very flexible tool, allowing us to “take stock” partway through the program and make some changes (as feasible) during the second part of the program. We then engage in a full empowerment evaluation at the end of the program, with participants using their experience in the program to plan for a “future” in which they will not take part but that will benefit the next group of students and teachers who participate. Over the years we have been able to implement and refine many of the suggestions generated by the empowerment evaluations, from more explicit training for scientist mentors to the inclusion of teacher “facilitators” to providing added opportunities for students to consider various STEM careers. The flexibility of empowerment evaluation techniques has resulted in a better experience for the students, teachers, scientists, and program coordinators each year of the program.
This week’s posts are sponsored by AEA’s Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group (http://comm.eval.org/EVAL/cpetig/Home/Default.aspx) as part of the CPE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting Collaborative, Participatory, and Empowerment Evaluations.
Cassie
You are too modest. I am very familiar with your work. You make and raise a number of interesting points in your posting. Hopefully I can expand on them a bit and shed some additional light on what I consider exemplary work.
Population – Constantly Changing Membership
You mention early on that empowerment evaluation is “an excellent way to make improvements to a program that has constantly changing membership.” Many of us are looking for evaluation tools and/or approaches with constantly changing memberships, since it is a context many of us work in, particularly with homeless, drug abuse, and runaway populations. Empowerment evaluation is typically associated with long-term capacity building efforts with the same population. This adds another dimension to our discussion about use and empowerment evaluation.
Capacity Building
Speaking of capacity building, I like the light-hearted way you interjected the role of capacity building: “the first year under the direct guidance of Dr. David Fetterman and subsequent years on our own (I guess that’s the capacity-building part!).”
This is actually a very important feature of empowerment evaluation. When I worked with Native Americans in Hewlett Packard’s $15 million Digital Village empowerment evaluation (http://www.davidfetterman.com/hewlettpackard.html) I follow these steps:
1. I began by facilitating some of the initial sessions informally in collaboration with a Native American colleague who had some knowledge of evaluation
2. Then I asked her to formally co-facilitate with me
3. After that, I asked her to lead the sessions and exercises with me standing (literally) in the background – so that their eyes were directed at her instead of me and so that I did not abdicate my responsibility by simply abandoning her. (I conferred with her before the session to strategize and after the session to debrief)
4. She took the lead and recruited another member of the tribe to facilitate the sessions after that (with me at a distance literally – but available as needed)
In some cases, folks can take responsibility immediately, in other cases you work together to ensure that it happens as rapidly as possible. I am certainly not a purist. When milked spilled on the floor at home when I was a kid, my father would say it doesn’t matter who spilled it, just clean it up. So if evaluation capacity is low then you do more than you think you need to until there is sufficient capacity to do the job – however – I believe it is your responsibility to provide training and apprenticeship opportunities to make sure that transition becomes a reality quickly.
Article
You should mention that after we worked together we published an article about it. While we respect community knowledge we also believe we have a contribution to make to the evidence-based foundation of scholarly knowledge. Here is a graph in the article that highlights part of the taking stock step of empowerment evaluation in that project.
In addition, here is the citation for those who would like additional detail concerning how this was applied and for those interested in experiential education.
Fetterman, D.M. and Bowman, C. (2001). Experiential Education and Empowerment Evaluation: Mars Rover Educational Program Case Example. Journal of Experiential Education. http://www.davidfetterman.com/mars.pdf
Tech-Orientation
Finally, I am not sure everyone noticed this sentence in your blog posting, but I certainly remembered this part: “In our case, online voting or survey software is used for both the voting and rating, and discussions are held via teleconference.” Much of your work is done electronically, as folks can see from one of your NASA teleconferencing sessions below:
Thanks again Cassie for letting folks know empowerment evaluation’s reach is literally for the stars – or at least the Mars rover. I am sure you had a great time at Harvard but it is probably time to return to Stanford (no pressure).
Take care.
-David
Hi Cassie:
Thanks for the very accessible description! I know I’ve worked with many programs that could benefit from this type of approach that might not have thought of engaging their short term participants so fully in the process!–Nicole
What a succint description of how easy it is to incorporate empowerment evaluation and the benefits it yields. In addition, who knew evaluation had made it to Mars.
Hi Cassie
Thanks for your good empowerment evaluation effort! To empower evaluation culture .I hopes have a chance to behave same active. Empowerment Evaluation is a phenomenon that spread in the four corner of the world. I think the essence of evaluation is empowerment. Empowerment Evaluation Approach has a grate position in the evaluation world. And I think Empowerment Evaluation currently pass from an evaluation approach and evolve an evaluation paradigm. For the reason that EE Is:
• Simple, small, understandable, & easy,
• Spacious, complete, holistic & innovator,
• Impulse, participate building & synergistic
My English blog from Saturday, March 01, 2003 (http://empowermentevaluation.blogspot.com/) attempt to have a role in dissemination of this best evaluation approach. You wrote: “For Step 2, after brainstorming a long list of program aspects/elements, those meeting in person are given 10 stickers that they use to “vote” on aspects of their program that are the most important by placing the stickers next to aspects on the list that they value.” I think only 5 stickers are enough?
Best wishes for you and all of evaluators and their stakeholders in our globe.
Moein