Hi, my name is Susan Geier and I am a doctoral student at Purdue University studying research methods, measurement and evaluation. I employ a participatory evaluation approach with the GEMscholar project and have learned much from the Native American college students and the dedicated program staff.

Lessons Learned: I would like to share my three R’s for participatory evaluation:

1. Build Rapport: In addition to conducting formal interviews and assessments, I interacted informally with the students and mentors when time allowed, during meals and in between activities. I spent time learning about Native American history and culture from the project team and students.

2. Demonstrate Relevance: I discussed with the stakeholders and participants possible benefits of the evaluation process and their unique roles in the improvement and success of the program components. For example, when the students expressed interest in helping future GEMscholars, a peer-mentoring option was added to the program. Consequently, students began to see the evaluation process as a mechanism for sharing their experiences and suggestions instead of an outside critique of their lives and activities.

3. Maintain Responsiveness: I provided the stakeholders with information in a timely and accessible format. Often these were oral reports followed by brief documents outlining the changes discussed. We had conversations about those issues that could not be resolved in a timely matter and possible effects on the program. In turn, the project team made ongoing changes, adding components where needed and modifying those elements that were not serving the objectives of the program. Assessments were modified if needed and the process continued.

Hot Tip: Journaling is a useful technique to capture real time reactions to interventions. This is particularly important when working with groups who are being introduced to unfamiliar and/or uncomfortable experiences as part of an intervention. I worked closely with the researcher and program coordinator to develop pertinent guiding questions for the students’ and mentors’ daily reflection journals. This is also a good time to develop an analysis rubric if applicable. Journals can be hand written or online (I provide a link to an online journal using Qualtrics). The journal entries provide a project team with valuable insights about how the program elements are perceived by all involved.

If you want to learn more from Susan, check out the Poster Exhibition on the program for Evaluation 2010, November 10-13 in San Antonio.

College Access Programs, Native American, participatory

Hello, I am Kimberly Kay Lopez, I have a community-based evaluation and research practice based in Houston, Texas. My work is concentrated in participatory evaluation methods used for the evaluation of youth programs and services.

Hot Tip: Using Photovoice in an Empowerment Evaluation: When working with youth, I have used the Empowerment Evaluation model many times. I have found that using Photovoice and journal writing within the Empowerment Evaluation model yields a variety of rich evaluation data. The goals of the Photovoice process enhance the evaluation process. The Photovoice process allows participants to document issues, engage in dialogue, and impact policy. I first integrated Photovoice with the Empowerment Evaluation model when evaluating the long-term impact of a multi-year teen pregnancy prevention program among urban Latino youth, I utilized the Photovoice process as a tool to “take stock” within the Empowerment Evaluation model. Youth were given cameras to capture the impact of the program. Youth were also given journals and guided writing assignments to express the impact that the teen pregnancy prevention program had on them. We also held traditional Empowerment Evaluation discussion groups. The youth and I analyzed visual data, journal data and discussion group data to develop the final evaluation report.

Rad Resource: PhotoVoice.org: PhotoVoice is an international organization that works with vulnerable populations. They offer several publications including a manual for using Photovoice. The methodology series gives further instruction on working with specific populations such as refugees. http://www.photovoice.org/shop/info/methodology-series

Rad Resource: PhotoVoice Manual: A comprehensive Photovoice Manual developed by Prairie Women’s Health Centre of Excellence. www.pwhce.ca/photovoice/pdf/Photovoice_Manual.pdf

Hot Tip: Using Journal Writing in Evaluations: I have found when working with middle school students, some students may be reluctant to participate in a discussion group. Offering youth writing opportunities allows those students uncomfortable in a group discussion a way to contribute to the evaluation process, on their terms.

Rad Resource: Guide on Engaging Youth: The National Clearinghouse for Families and Youth has a great guide on engaging youth in writing. www.ncfy.com/publications/pdf/lbd_write.pdf

There are all kinds of ways to get creative with data collection-digital storytelling, video cameras, blogs, tweets, text messages! Get creative! Use your imagination! Have fun!

Want to learn more from Kimberly? She’ll be on the program this November at Evaluation 2010, The American Evaluation Association’s Annual Conference in San Antonio, Texas.

community, journal, participatory, photovoice

I’m Jim Burdine and a few weeks ago I posted a “tip-a-day” (see post here) on presenting community health assessment data back to the community. This is part two of that discussion. Last time I focused on “what to present” this posting focuses on “how to present . . . with results.”

There are two goals for any assessment presentation: (1) reporting the data, and (2) identifying next steps for improving health status. I’ve used a format that is effective in a variety of settings. This assumes, as discussed in my last posting, that the assessment has been a “participatory process.”

Hot Tip: Hold a “regional health summit.” Folks invited include those who have participated in various aspects of the assessment as well as others representing a broad cross section of community sectors. My experience has largely been in medium to smaller communities (500K to 10K population) but participation is roughly the same – about 150-250 folks is a good turnout. The event is split into two parts and can be accomplished in one day or two. The two parts are a presentation of the data, followed by a workshop. Ideally the data presentation is limited to an hour and the actual presentation, or much of it, is by a community member. Avoid the temptation to have more than a couple of speakers – “inclusiveness” is a nice idea but counterproductive in this context. You can invite community members to do “welcomes” and other remarks.

Following the data presentation is another opportunity to develop for community buy-in. In the workshop, we ask folks to select among the “findings” of the assessment (e.g., “access to mental health,” “childhood obesity,” “rural transportation”) and sign up for those topics. Have rooms/tables setup for each group. A convener appointed for each group opens the discussion and is responsible for encouraging folks to commit to participating in an ad hoc task group on this topic that that would agree to meet 2-3 times over the next 2-3 months. At the end of that time (3 months) the larger group is reconvened to hear reports from the various ad hoc task groups and then to endorse a regional health improvement strategy developed at that session (based on the group reports). During the 2-3 month work period, each team reviews pertinent findings and identifies “best practices” of interventions related to that issue. The group then identifies one or two priority activities that incorporate those best practices as adapted to your community.

By following this model, what might have been just a data presentation yields a comprehensive regional strategy with multiple sub-parts that is endorsed by the entire community (at least as represented by the individuals and agencies participating in the overall process).

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

communication, participatory

Hello fellow evaluators!! My name is Kim Sabo Flores and I am the creator of Evaluation Access, an evaluation capacity building organization that serves non-profit staff and youth participants.  I began my evaluation career with a focus on engaging children and youth in evaluation but soon realized that adult staff also needed support.  Over the years, it is the young people who have taught me how to truly put evaluation into the hands of real people.

Hot Tip: PLAY!!!  Playing with evaluation concepts and methods helps level the playing field so that staff and youth can begin to see evaluation as something that everyone can do. By playing with evaluation it is possible to invite staff and youth to participate in creating evaluations that more appropriately reflect their work’s processes and outcomes. I often begin playing with clients (both youth and adults) through storyboard logic models. These stories generally include three to four cells: cell #1 what is going on for you client, constituent group, or community before they join your program; cell #2 what is going on for your client once they begin your program; cell #3 what is going on for your client after they leave your program.   I then have them create their logic models using photographs, cartoons, crayons, markers, sticker, etc. (see my book Youth Participatory Evaluation: Strategies for engaging young people for more instructions).

Rad Resource: Most recently, I have been using “bit strips” www.bitstrips.com as a really cool resource for helping both youth and adults create their storyboard logic models.  Bitstrips is a new online comic strip technology that has numerous open access functions.

Hot Tip: Hire a young person to work closely with you.  Empower these young people to help you think about how to make evaluation fun and interesting.  I guarantee that it will change your practice, and at a minimum, it will help you dive more deeply into web 2.0 technologies.

Rad Resource: One of the young people I hired taught me about the value of “starburst” for likert scale responses.  For example, at the end of a workshop with young people, you can ask them to use the candy to rank their experiences.  For example: you can ask them: How much did you learn during this workshop (Nothing, A little, Some, A lot). As they leave the room have the question posted up on the wall with a bucket underneath. Have them drop a red starburst in the bucket if they learned “a lot”, a green if they learned “a little”, and a yellow if they learned “nothing”.  I generally post about three or four question and have three or four buckets positioned under each. Now all of my workshop assessments include candy. Starbursts are really special because they come individually wrapped (cuts down on the mess), they come in five colors, and as data goes they are pretty tasty.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

engagement, participatory, youth

Hi, my name is Sophie Alvarez and I work in the Challenge Programme on Water and Food (CPWF) as a consultant in monitoring and evaluation. During Phase I of the CPWF, we began developing an evaluation and monitoring methodology where a group of project researchers and some of their stakeholders get together and map the project’s expected ‘impact pathways’: the Participatory Impact Pathways Analysis (PIPA)

We have been enriching PIPA through using it in varied settings. PIPA is methodology that draws from the fields of program theory evaluation, social network analysis and research to understand and foster innovation. PIPA begins with a workshop in which project or program leaders and stakeholders get together and ‘map out’ the pathways they think the project will follow to achieve impact, and who they will need to work with to move along successfully down their pathways. Participants of the workshop work on agreeing on a vision, discuss and draw maps of the partnerships they will work in, and make explicit their theories of how the project will function: from activities to outcomes and impact. These mapped ‘outcome pathways’ and ‘theories of change (TOC)’ become the project’s ‘blueprints’ that can then be monitored by visiting and adjusting them periodically, and which can also be used for ex-post impact evaluation.

PIPA- thinking encourages project leaders to think beyond the scope of a single project, and it requires that project interested parties make explicit and discuss what it is they want to do, and how.  PIPA engages stakeholders in a structured participatory process, promoting learning and understanding of issues such as scaling up and out networks (the way the project’s innovations get adopted and mainstreamed at local, national and international levels), adaptive management and monitoring and evaluation for learning. Project Leaders also derive outcome targets and milestones which are regularly revisited and revised as part of project monitoring and evaluation (M&E).

Hot Tips:

• The PIPA workshop works best with a group of 4-5 projects, with 3-4 participants from each. It also works well when you invite final users, policy makers and other actors who are interested in the outcomes of your project.
• Get projects thinking on their activities to outputs to outcomes and impact routes. Remind them that their project’s influence doesn’t finish when the project does!
• Use all the group dynamics and facilitation powers to get people to really discuss and go beyond platitudes: we incorporate in our workshops many of the tools and methods found in this Knowledge Sharing Toolkit. http://www.kstoolkit.org/

You can go here to see our PIPA publications, and for more references.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

impact, participatory, pathway analysis

Molly Engle on Evaluation as an Everyday Activity

Hello. I am Molly Engle, the Evaluation Specialist for the Oregon State University Extension Service and a past president of the AEA. For over 10 years my role has been building evaluation capacity and providing evaluation resources to over 250 faculty members. Although these individuals are well educated in their content expertise covering a wide range of community-based topics (such as obesity, physical activity, silviculture, invasive species, integrated pest management, cereal crops, youth development, incarcerated youth, among others), they are not evaluators. Consequently, evaluation feels monumental to them. I have developed the following approach to decreasing that feeling.

Hot Tip: I believe that evaluation is an everyday activity. If that is so (which I believe), then why does it FEEL so monumental–you know–over whelming, daunting, aversive even? Evaluation to be effective must be manageable, workable, and useful.

I can think of several reasons for that feeling:

• You don’t know how.
• You don’t want to (do evaluation).
• You have too much else to do.
• You don’t like to (do evaluation).
• Evaluation isn’t important.
• Evaluation limits your passion for your program.

All those are good reasons. Yet, in today’s world you have to show your programs are making a difference. You have to provide evidence of impact. To do that (show impact, making a difference) you must evaluate your program.

Rad Resource: My evaluation blog, Evaluation is an everyday activity, helps make evaluation manageable because it looks at evaluation in small amounts.

Hot Tip: How do you make your evaluation manageable? How do you make it an everyday activity? Here are several ways.

• Set boundaries around what you evaluate.
• Limit the questions to ones you must know. Michael Patton says only collect data you are going to use, then use it.
• Evaluate key programs, not every program you conduct.
• Identify where your passion lies and focus your evaluation efforts there.
• Start small. You probably won’t be able to demonstrate that your program ensured world peace; you will be able to know that your target audience has made an important change in the desired direction.

Rad Resource: To read more about evaluation and use, see Michael Quinn Patton’s book, Utilization-Focused Evaluation. His new 4^th edition is awesome.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

fear of evaluation, participatory, stakeholders, use

My name is Ann Zukoski and I am a Senior Research Associate at Rainbow Research, Inc. in Minneapolis, Minnesota.

Founded as a nonprofit organization in 1974, Rainbow Research’s mission is to improve the effectiveness of socially-concerned organizations through capacity building, research, and evaluation. Projects range in scale from one-time program evaluations to multi-year, multi-site research studies and designs that explicitly include participatory approaches designed to lead to program improvement.

Through my work, I am always looking for creative ways to capture evaluation data. Here is one rad resource and a hot tip on a participatory tool to add to your tool box.

Rad Resource: Participatory evaluation approaches are used extensively by international development organizations. This web page is a great resource for exploring different rapid appraisal methods that can be adapted to the US context.

ELDIS – http://www.eldis.org/go/topics/resource-guides/participation/participation-in-development – ELDIS provides descriptions and links to a variety of information sources on participatory evaluation approaches, including online documents, organization’s web sites, databases, library catalogues, bibliographies, and email discussion lists, research project information, map and newspaper collections. Eldis is hosted by the Institute of Development Studies in Sussex, U.K.

Hot Tip: Evaluators are often asked to identify program impacts and measure key outcomes of community based projects. Impact and outcome measures are often externally determined by the funder. Many times, however, collaborative projects lead to unanticipated outcomes that are seen to be of great value by program participants but are overlooked by formal evaluation designs. One participatory technique, Most Significant Change (MSC), offers an alternative approach to address this issue and can be used to surface promising practices.

Most Significant Change Technique (MSC) – MSC is a participatory qualitative data collection process that uses stories to identify the impact of the program. This approach involves a series of steps where stakeholders search for significant program outcomes and deliberate on the value of these outcomes in a systematic and transparent manner. Stakeholders are asked to write stories of what they see as “significant change” and then dialogue with others to select stories of most importance. The goal of the process is to make explicit what stakeholders (program staff, program beneficiaries and others) value as significant change. The process allows participants to gain a clearer understanding of what is and what is not being achieved. The process can be used for program improvement, identifying promising practices as well as to uncover key outcomes by helping evaluators identify areas of change that warrant additional description and measurement.

Where to go for more information:

http://www.mande.co.uk/docs/MSCGuide.pdf

http://www.healthcomms.org/comms/eval/le02.html

Have you used this tool? Let us all know your thoughts!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

international, Most Significant Change, participatory, stakeholders

My name is Laura Myerchin Sklaroff and I am a Project Manager, Researcher, and Evaluator with Los Angeles County Department of Health Services, Clinical Resource Management (LAC DHS CRM). My focus is on health care quality improvement for low income and underrepresented populations; prior to LAC DHS CRM I coordinated the nationally recognized practice-based research network, LA Net, and I have worked on several federally and foundation funded national and regional studies. I also teach a graduate level Evaluation Procedures class at the Tseng College, California State University, Northridge.

I frequently use components of Community Based Participatory Research (CBPR), also known as Participatory Action Research (PAR), when conducting health care research and evaluations. CBPR is based on the idea that research should include the following principles (Isreal, 2000; full citation is in the Rad Resources section):

• Respects and Recognizes Community as a Unity of Identity
• Builds on Strengths and Resources in the Community
• Facilitates Collaborative, Equitable Involvement of All Partners in All Phases of Research
• Integrates Knowledge and Intervention for Mutual Benefit of All Members
• Information and Learning is Shared Equally by All Members
• Involves a Cyclical and Iterative Process
• Addresses Research Topics from Both Positive and Ecological Perspectives
• Disseminates Findings and Knowledge Gained to All Partners
• Involves a Long-Term Commitment by All Partners

While similar to several themes found in many evaluation theories, I’ve found that outlining the principles of CBPR for stakeholders at the start of the evaluation or research process greatly increases participant by-in. As I review the process to stakeholders, I make it a point to describe how my actions integrate and are consistant with the CBPR principles. When conducting health care evaluations, particularly with stakeholder groups that are historically distrustful of medical research, the CBPR principles offer an excellent standard for stakeholders to use when “evaluating the evaluator.”

For the evaluator, CBPR offers language that can be very helpful in drafting memorandums of understanding (MOUs), Research Collaboration Agreements, and Mission and Value Statements. Exploring CBPR literature is also a great way to understand the ins and outs that come with doing “bottom-up” research.

Rad Resources: Here are a few resources.

• Isreal, Barbara A. “Community-Based Participatory Research: Principles, Rationale and Policy Recommendations.” Successful Models of Community-Based Participatory Research, pp. 16-22, March 2000, Washington, DC.
• Developing and Sustaining Community-Based Participatory Research Partnerships: A Skill-Building Curriculum (http://www.cbprcurriculum.info/)
• The Community-Campus Partnerships for Health (CCPH) page: http://depts.washington.edu/ccph/index.html
• The Academic Autism Spectrum Partnership in Research and Education (AASPIRE) about CBPR page: http://www.aaspire.org/about/cbpr.html

And the CBPR listserv is great resource for discussion, sharing of ideas, and collaborative problem-solving! They also offer information about conferences, funding opportunities, job announcements, and new publications relevant to CBPR. You can join this listserv here: http://mailman2.u.washington.edu/mailman/listinfo/cbpr

Happy Evaluating!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

cbpr, community, participatory

My name is Cindy J Wong. I am a consultant evaluation and a social science researcher in health and human services. Recently, I have been obsessed with digital video technology as a tool in social, organizational, instructional, and evaluative documentation. I recently had the chance of bringing portable video recorders to a visit to a non-profit organization located in South Africa. The organization provides mobile health services and computer education at primary schools in rural areas to address HIV/AIDS. I had an idea about how the technology might be utilized, but I was pleasantly surprised and thrilled, as the members of the organization had more immediate ideas for the technology that involved educational assessment and implemented them. The staff members are continuing to expand the ways in which the technology is utilized in education and instruction.

Rad Resource: Portable video recording devices, such as FlipVideo (http://www.theflip.com/en-us/) record up to 120 minutes of high definition video. These are hand-held battery-operated units that contain retractable USB ports which can be plugged directly into a computer or laptop for download. The cameras include basic software for downloading, organizing and editing of the video clips. The cameras are affordable as the technology goes, and they are widely available in the United States. Movie Maker 2.1 (http://bit.ly/moviemaker2) has been a powerful software program that can integrate with the Flip system. As a Windows User, you may not even realize that this software is on your computer, since it is downloaded through automatic updates (check your Program Folder). MacUsers can similarly use iMovieMaker (http://www.apple.com/ilife/imovie/). File conversion freeware such as Pazera (http://bit.ly/pazera) convert Flip HD file formats to Windows Movie Maker formats with ease.

I hope you enjoyed this Rad Resource, Cheers!

This week’s posts are sponsored by AEA’s Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group (http://comm.eval.org/EVAL/cpetig/Home/Default.aspx) as part of the CPE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting Collaborative, Participatory, and Empowerment Evaluations.

empowerment, participatory, technology

My name is Daniel Brandão and I am an evaluator from Brazil. I work at Instituto Fonte as a consultant and we have a participatory approach to evaluation. I used to work with evaluation of programs focused in youngsters under social exclusion situations where methods oriented by arts were very useful. In 2007 to face the challenge of creating a good environment to have in-depth conversations with youngsters in conflict with the law we developed the FRAMES Method. Inspired by comics, FRAMES is a set of 27 drawings used to trigger dialogues with these youngsters.

Rad resource: You can get the complete set of drawings at: http://tinyurl.com/ygqafsk. We didn’t produce any article or so about FRAMES. But it is in our 2010 to-do list! A good resource about arts based research practice is the book “When Method Meets Art” (Levy, P. Guilford Press)

FRAMES

• Can be adjusted for multiple possibilities of applications.
• Can be used in evaluations or as an educational tool.
• Can be applied Individually or with groups.
• With both genders
• With youngsters over 12 year old
• Used by psychologists, social service assistants, non formal educators, social scientists and others.
• Application time lasted from 20 minutes to 5 hours.

The linked document is written in Portugese, but here’s a link to an online translation program: http://www.freetranslation.com/

Regards!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

art, dialogue, instrument, participatory, youth