I’m Michael Matteson and I’m pursuing a graduate degree at the University of Wollongong (Australia).

Lessons Learned – What’s the Hawthorne effect? Why would it matter in Empowerment Evaluation? Most people will remember hearing of the “Hawthorne effect “based on a major study of organizational and environmental effects on productivity which gave confusing and contradictory results over a long period. Commentators suggested that any effects of the experiments were a result, not of the researcher’s manipulation of variables, but of staff feeling important because they were being observed. Stephen Draper gives a definition of the Hawthorne effect that I find useful:

An experimental effect in the direction expected but not for the reason expected; i.e. a significant positive effect that turns out to have no causal basis in the theoretical motivation for the intervention, but is apparently due to the effect on the participants of knowing themselves to be studied in conjunction with the outcomes measured (Draper, 2009).

Looking at this in terms of Empowerment Evaluation, I’ve come to feel that the evaluation team’s experience of the evaluator is a major part of their experience of the evaluation. This makes it a legitimate part of the process use of the evaluation, which is the mechanism expected to enable the empowerment result.

If so, it’s important to clarify what the effect is in each situation. This will depend on what the evaluator is doing, including the atmosphere they’re providing, and the extent to which the evaluator’s involvement is part of the positive reinforcement that team members’ experience, along with their own decision-making, in the course of the evaluation.

The Hawthorne effect can be expected to modify results outside of the conscious parameters of the investigation unless consciously allowed for. In the case of data gathering, I have decided to combine observation of classes run by staff who were part of the evaluation team with observation of classes that weren’t, hoping that any Hawthorne effect in the result-gathering will be canceled out by a parallel Hawthorne effect in my observation of the non-participants.

Hot Tip: Impression management, common in focus groups, and based on Irving Goffman’s work, may be relevant here.

Resource: Draper’s article explains the issues involved and the many uses of the Hawthorne experiment’s continuing legacy. Draper, S.W. (2009, Dec 23) The Hawthorne, Pygmalion, Placebo and other effects of expectation: some notes

Lessons Learned: While the most common explanation of the Hawthorne experience is some kind of “people felt important” effect, Paul Blumberg’s 1969 Industrial Democracy: The Sociology of Participation (Schocken) already argued, based on the original research, that the most likely factor was the level of the participants’ involvement in decision-making. This aspect has been consistently ignored.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

empowerment, Hawthorne Effect

My name is Aimee Sickels and I am the Principal Evaluator and owner of Custom Evaluation Services, an independent evaluation consulting business. I am also currently working on a Ph.D. in Social Work at the University of South Carolina. My helpful hint relates to the use of logic models. I am an empowerment evaluator and believe that our role as evaluators is not only to evaluate the programs, but to build the evaluation capacity in the organizations we evaluate. So, how can the logic model help?

Hot Tip: The 2 minute logic model. I will often look over my client’s programming before I am going on a site visit or a monitoring meeting. I will select a single program item, it may be a single goal or a single activity, and I will bring this to my client as the 2 minute logic model. Before we get started with our planned meeting I present my single item to them, for example one of my client’s who serve vulnerable persons has a goal of actually increasing involvement of the client in the program design; having the client’s participate more in their own service design. I start this on the input column and walk the client through that single item across the output column and ultimately the outcome column. I then say spend the rest of today and the rest of this week working on this single item. Because it is a single item and a single line it is easy to follow and quick to see and understand. Put the 2 minute logic model write up at each staff person’s desk and ask them to consider it with each client they see. I can guarantee that they will see an improvement on that line of programming for that week! Try the 2 minute logic model today!

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources. You can also learn more from the DOVP TIG via their many sessions at Evaluation 2010 this November in San Antonio.

empowerment, logic models

Hi my name is Melissa Rivera. I am the Director of Evaluation and Research at the National Center for Prevention and Research Solutions (NCPRS). Since 2006, our organization has collaborated with the National Guard Bureau to implement an evidence-based program, Stay on Track, to over 115,000 sixth through eighth grade students nationally. Since inception, NCPRS has strategically aligned our evaluation goals with the National Guard Counterdrug Program’s goals. Over the years we have learned several best practices that have helped us engage stakeholders throughout the evaluation cycle.

Hot Tip: A successful evaluation design requires engaging and empowering stakeholders throughout the evaluation cycle. We developed a comprehensive evaluation plan that reinforced and aligned:

• the goals and objectives of stakeholders
• the goals of national organizations
• programmatic objectives

To reinforce messaging, we ensured that training workshops and any materials developed consistently met identified goals.

Rad Resource: Rosalie Torres, Hallie Preskills, and Mary Piontek have created an effective tool that can be used to engage the stakeholders in evaluation strategies. In their book, Evaluation Strategies for Communicating and Reporting: Enhancing Learning in Organizations, they provide explicit details on how to develop a communication and reporting plan that can be used to engage stakeholders.

We have used these strategies and incorporated them into our evaluation plan and the results are promising. This information is also discussed during our training workshops and has provided reinforcement of our goals and objectives.

Hot Tip: Develop interactive multi-tier trainings that engage implementers and provide them with the tools they need to implement effectively and incorporate a certification element that enables them to train others. Some best practices include providing implementers with:

• the knowledge and resources they need to cascade the information appropriately.
• a procedural guide or a guidance document that contains an overview of the program, the design of the evaluation, how to administer surveys, how to return surveys, and a glossary of evaluation terms.

Collectively, engaging stakeholders equates to success in every facet of evaluation.

We hope that you consider using some of these tools!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

empowerment, engagement, stakeholders

My name is Kim Norris and I am the Evaluation Coordinator for University of Maryland Extension’s Food Supplement Nutrition Education (FSNE) program. Included in my work is to assist educators in developing useful strategies for assessing the impact of their work on our target audience, limited-income, and often, low-literacy, populations.

Hot Tip: Utilize Audience Response Technology System in group class settings for immediate, anonymous assessment and feedback.  The “clicker” technology, as we sometimes refer to it, allows questions to be asked both orally and in writing, allows individuals to respond anonymously, and allows for immediate feedback after responses for both educator and audience.  We used these recently with our own educators to ask questions that, in a setting in which anonymity was not guaranteed, could lead to false answers due to high motivation to fall within socially acceptable norms.  Since results are calculated and visible to all on the spot, group responses can be reviewed, analyzed, interpreted, and addressed by the group, thereby increasing potential for empowering participants.

Other advantages of using this easy-to-teach technology include the ability to:

• collect data from larger numbers of people in a shorter amount of time
• eliminate data entry errors by direct transfer of electronic data to a data base
• engage technology-averse populations in computer technology to their benefit
• help low-literacy populations participate in surveys as respondents
• provide confidentiality for respondents

The technology can lead to missing data if not preceded by sample questions and as group sizes become larger or less engaged.  Studies are underway to better understand strengths and limitations of the technology as an educational and evaluation tool.

Rad Resource: A Bibliography of Selected Readings on Audience Response Systems: http://bit.ly/audienceresponsesystems.

This week’s posts are sponsored by AEA’s Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group (http://comm.eval.org/EVAL/cpetig/Home/Default.aspx) as part of the CPE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting Collaborative, Participatory, and Empowerment Evaluations.

collaboration, empowerment, participation, technology

My name is Cindy J Wong. I am a consultant evaluation and a social science researcher in health and human services. Recently, I have been obsessed with digital video technology as a tool in social, organizational, instructional, and evaluative documentation. I recently had the chance of bringing portable video recorders to a visit to a non-profit organization located in South Africa. The organization provides mobile health services and computer education at primary schools in rural areas to address HIV/AIDS. I had an idea about how the technology might be utilized, but I was pleasantly surprised and thrilled, as the members of the organization had more immediate ideas for the technology that involved educational assessment and implemented them. The staff members are continuing to expand the ways in which the technology is utilized in education and instruction.

Rad Resource: Portable video recording devices, such as FlipVideo (http://www.theflip.com/en-us/) record up to 120 minutes of high definition video. These are hand-held battery-operated units that contain retractable USB ports which can be plugged directly into a computer or laptop for download. The cameras include basic software for downloading, organizing and editing of the video clips. The cameras are affordable as the technology goes, and they are widely available in the United States. Movie Maker 2.1 (http://bit.ly/moviemaker2) has been a powerful software program that can integrate with the Flip system. As a Windows User, you may not even realize that this software is on your computer, since it is downloaded through automatic updates (check your Program Folder). MacUsers can similarly use iMovieMaker (http://www.apple.com/ilife/imovie/). File conversion freeware such as Pazera (http://bit.ly/pazera) convert Flip HD file formats to Windows Movie Maker formats with ease.

I hope you enjoyed this Rad Resource, Cheers!

This week’s posts are sponsored by AEA’s Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group (http://comm.eval.org/EVAL/cpetig/Home/Default.aspx) as part of the CPE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting Collaborative, Participatory, and Empowerment Evaluations.

empowerment, participatory, technology

Greetings!  We are Tom McQuiston (USW Tony Mazzocchi Center) and Tobi Mae Lippin and Kristin Bradley-Bull (New Perspectives Consulting Group).  We have collaborated for over a decade on participatory evaluation and assessment projects for the United Steelworkers (labor union).  And we have grappled mightily with how to complete high-quality data analysis and interpretation in participatory ways.

Hot Tip: Carefully determine up front what degree of full evaluation team participation there will be in data analysis.  Some practical considerations include:  the amount of team time, energy, interest, and analysis expertise that is available; the levels of data analysis being completed; the degree of project focus on team capacity-building; and the project budget and timeline.  How these and other considerations get weighed is, of course, also a product of the values undergirding your work and the project.

Hot Tip: Consider preparing an intermediate data report (a.k.a. “half-baked” report) that streamlines the analysis process for the full team.  Before the full team dives in, we:  review the raw quantitative data; run preliminary cross-tabs and statistical tests; refine the data report content to include only the — to us — most noteworthy data; remove extraneous columns spit out of SPSS; and assemble the tables that should be analyzed together — along with relevant qualitative data — into reasonably-sized thematic chunks for the team.

Hot Tip: Team time is a precious commodity, so well-planned analysis/ interpretation meetings are essential.  Some keys to success include:

1. Invest in building the capacity of all team members.  We do this through a reciprocal process of us training other team members in, say, reading a frequency or cross-tab table or coding qualitative data and of them training us in the realities of what we are all studying.
2. Determine time- and complexity-equivalent analyses that sub-teams can work on simultaneously.  Plan to have the full team thoughtfully review sub-team work.
3. Stay open to shifting in response to the team’s expertise and needs.  An empowered team will guide the process in ever-evolving ways.

Some examples of tools we have developed — yes, you, too, can use Legos™ in your work — can be found at: http://newperspectivesinc.org/resources.

We never fail to have many moments of “a-ha,” “what now” and “wow” in each participatory process.  We wish the same for you.

This week’s posts are sponsored by AEA’s Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group (http://comm.eval.org/EVAL/cpetig/Home/Default.aspx) as part of the CPE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting Collaborative, Participatory, and Empowerment Evaluations.

analysis, collaboration, empowerment, participation

My name is Wayne Miller and I am a senior lecturer in the Faculty of Education at Avondale College, Lake Macquarie, New South Wales, Australia.  In December I received my doctorate from the University of Wollongong following acceptance of my thesis titled Practical methods to evaluate school breakfast programs – A case study.   The study reports the use of empowerment evaluation with a national school breakfast program in Australia known as the Good Start Breakfast Club (GSBC).

During the project some eighty GSBC program personnel took part in ten empowerment evaluation workshops to identify key program activities for investigation; gather baseline data about the strengths and weaknesses of the activities; suggest goals and strategies to monitor and improve the activities identified; and to develop evaluation tools designed to provide evidence of success.  Following workshops I asked participants …from your experiences in these initial workshops how valuable do you think the empowerment evaluation method is for collaboratively evaluating the GSBC program?  42/80 indicated ‘very’ to ‘extremely’ valuable with a further 36/80 responding ‘reasonably’ to ‘quite’ valuable.  A regional coordinator commented, the model is definitely in line with the principles of our program and empowering the community.  One beautiful response from an outlier, a total waste of time and all about Miller getting his doctorate!

Toward the end of the project I interviewed 29 program personnel who had been directly involved in the evaluation and I asked them to reflect on the empowerment evaluation process and particularly whether it had adhered to the ten principles of empowerment evaluation.  Respondents made up of volunteers and teaching staff at the ‘coal face’, school principals, GSBC coordinators and executive staff from Red Cross the program managers, and the Sanitarium Health Food Company, the major sponsor, reported both alignment and misalignment with the principles.  Two examples: On the principle of democratic participation defined as active participation by everyone in shared decision-making is valued…respondents acknowledged that the ‘taking stock’ step of the empowerment evaluation had been particularly democratic but that the democratic nature of the evaluation process had been compromised when those who came together to implement Step 3 – ‘Planning for the future’ were handed evaluands from workshop groups who had completed Steps 1 – Develop a mission, vision or unifying purpose for the program and Step 2 -Taking stock.   On the principle of capacity building defined as program staff and participants learn how to conduct their own evaluations…significant gains in evaluation capacity was reported by personnel at the breakfast club level.  Volunteer staff at one site designed and trialled an instrument which provided average nutrient uptake data which were subsequently used to modify food served to improve fibre intake.  A negative aspect reported was that staff turnover at management level mitigated against evaluation capacity building in one region.

Hot Tip: Trustworthy relationships must be established for empowerment to occur.  Community ‘champions’ committed to their communities, who use empowering processes and have good networks and communications skills, are vital partners.  Ongoing commitment by senior management to nurture and support empowered staff by providing them with the resources necessary to remain so, is a ‘must have’ ingredient to avoid empowerment fade.

This week’s posts are sponsored by AEA’s Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group (http://comm.eval.org/EVAL/cpetig/Home/Default.aspx) as part of the CPE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting Collaborative, Participatory, and Empowerment Evaluations.

collaboration, empowerment, trust

My name is Linda F. Delaney and I serve as a program evaluator for my own company (LFD Consulting, LLC) in Marion, Arkansas.  Since 2005, I have worked closed with Dr. David Fetterman as an evaluator for the Minority Sub-recipient Grant Office at the University of Arkansas in Pine Bluff, Arkansas (UAPB).  I also serve as President of the Arkansas Group of Evaluators (AGEs).  I currently conduct program evaluations for 5 local tobacco control organizations in Arkansas.  These agencies implement tobacco control efforts in approximately 20 of Arkansas’ 75 counties.   The nature of this work requires collaboration through empowerment strategies to increase effectiveness.  These tobacco program coordinators must be prepared to function in interdisciplinary groups ranging from the medical/dental profession to government.  It is important that evaluators empower their clients through training them in empowerment evaluation and effective collaboration.  I will be sharing some hot tips about effective training methods.

Hot Tip: Strategically begin with the clarity and the coordination of primary stakeholder missions.  Are they congruent?  Allow the clients to share the collaborative purposes (first).  Remember, what they say is what you hold them accountable for – so ask pertinent questions that solicit the answers needed to drive the conversation in the direction you know that it must go. (Principle:  If I say it, you may doubt it.  If you say it, you believe it – and that is what you are held accountable for.)

Hot Tip: Take stock by finding out who is currently involved in the activities to complete goals and objectives.  Then brainstorm who else can be involved and why others are needed.  Make this conversation in depth.  Determine various technological tools that can help communication and partnering efforts.  (Principle:  If one understands the WHYs, the HOWs will take care of themselves.)

Hot Tip: Develop a bullet-point strategy/plan with deadlines for recruitment and implementation to involve and empower collaborative partners needed to complete planned activities. (Principle:  Together Everyone Achieves More)

Rad Resource:  http://tobaccoprevention.blogspot.com/ is a blogspot where you can find many tips and strategies on collaboration through empowerment.

This week’s posts are sponsored by AEA’s Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group (http://comm.eval.org/EVAL/cpetig/Home/Default.aspx) as part of the CPE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting Collaborative, Participatory, and Empowerment Evaluations.

collaboration, empowerment, participation

My name is Cassie Bowman. As the coordinator of NASA’s Mars student intern program, I’ve looked for ways to continuously improve the experience for the student, teacher, and scientist participants. My tip is that empowerment evaluation, used in a modified format, can be an excellent way to make improvements to a program that has constantly changing membership.

Though empowerment evaluation is most commonly used to build capacity and foster improvement among stakeholders and staff in a group or organization, NASA’s Mars Public Engagement Program has found it can be used very successfully to evaluate and improve a program with changing group membership. We have used empowerment evaluation in our Mars student intern program with each year’s group of students and teachers—the first year under the direct guidance of Dr. David Fetterman and subsequent years on our own (I guess that’s the capacity-building part!).

The standard model of empowerment evaluation involves three main steps (Fetterman, 2001):

• Step 1: “Developing a mission, vision, or unifying purpose”
• Step 2: “Taking stock or determining where the program stands, including strengths and weaknesses”
• Step 3: “Planning for the future by establishing goals and helping participants determine their own strategies to accomplish program goals and objectives.”

Though it sounds highly conceptual, these steps are carried out in very concrete ways. For Step 1, if the group engaged in the evaluation is meeting face-to-face, they might go about drafting (or editing) a mission/vision/unifying purpose together in real-time. In our case, because our student/teacher groups are distributed across the country, we have people submit their ideas electronically, then share drafts back and forth before coming to consensus via teleconference. For Step 2, after brainstorming a long list of program aspects/elements, those meeting in person are given 10 stickers that they use to “vote” on aspects of their program that are the most important by placing the stickers next to aspects on the list that they value. Once the list is whittled down to a reasonable number through this voting process, participants rate each aspect (usually on a scale from 1-10) and then discuss the ratings. In our case, online voting or survey software is used for both the voting and rating, and discussions are held via teleconference. Finally, as with the previous steps, Step 3, planning for the future, can be held in person or via distance technologies.

We have found empowerment evaluation to be a very flexible tool, allowing us to “take stock” partway through the program and make some changes (as feasible) during the second part of the program. We then engage in a full empowerment evaluation at the end of the program, with participants using their experience in the program to plan for a “future” in which they will not take part but that will benefit the next group of students and teachers who participate. Over the years we have been able to implement and refine many of the suggestions generated by the empowerment evaluations, from more explicit training for scientist mentors to the inclusion of teacher “facilitators” to providing added opportunities for students to consider various STEM careers. The flexibility of empowerment evaluation techniques has resulted in a better experience for the students, teachers, scientists, and program coordinators each year of the program.

This week’s posts are sponsored by AEA’s Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group (http://comm.eval.org/EVAL/cpetig/Home/Default.aspx) as part of the CPE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting Collaborative, Participatory, and Empowerment Evaluations.

collaboration, empowerment