Greetings! My name is kas aruskevich and I am principal of Evaluation Research Associates LLC. I live in Fairbanks and work primarily in rural Alaska. Alaska is known for its great natural beauty, extreme temperatures, and unique context of diverse and far-flung communities assessable only by air. Alaska is the largest state in the U.S.

Rural communities often have a small population and rarely have a local evaluator for hire. Consequently, a program evaluator is most often hired from outside the community or region. Helicopter evaluation is a depreciating term used to describe a drop in – evaluate – depart approach. Today’s post talks about methods to strengthen and add depth to evaluations that involve distance between evaluator and evaluand.

Hot Tip: First, context is important. Familiarize yourself with the community and region before you travel. Gather demographic data of the community, leading industry, and cultural composition. Learn about the organization hosting the program, before your first contact. Plan your site-visit around a community event so you can see the community in a broader context.

Rad Resource: The importance of context is discussed in New Directions for Evaluation Fall 2012, Issue 135.

Hot Tip: Next, work to build open communication with program staff. Begin with a teleconference to provide an opportunity to meet staff and organization and discuss program status. Teleconferences also give you a chance to describe your evaluation style and see if you are a ‘fit’ for the organization and the evaluation project.

ALWAYS include participatory methods. I don’t ‘come in’ as the expert with an unchangeable evaluation design, but instead write up suggestions for the evaluation to negotiate before a plan is finalized. As an itinerant evaluator you can’t be on site as often as you might like. Using a participatory evaluation approach, program staff can be involved in the evaluation through taking photos or identifying program participants or stakeholders to interview.

Rad Resource – Read more about participatory evaluation in Cousins and Chouinard’s new book Participatory Evaluation Up Close.

Hot Tip: Lastly, work to build a friendly relationship based on mutual interests with at least one person in the organization or community. After years of conducting evaluations, friendly relationships have evolved into continuing friendships. These friendships have mutual benefits, in-part, they are a bridge for the evaluator to learn community specific cultural protocols–very important to conduct evaluations in cross-cultural settings – which in turn can strengthen the program through appropriate evaluation.

Lesson Learned: Itinerant evaluation can be much more than a helicopter site-visit approach. Regular communication and working together with program staff as a team can expand the evaluative evidence collected and increase report credibility, relevance, and use by the program staff.

The American Evaluation Association is celebrating Alaska Evaluation Network (AKEN) Affiliate Week. The contributions all this week to aea365 come from AKEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

affiliates · participatory · rural

I am Susan Kistler, the American Evaluation Association’s Executive Director and aea365’s regular Saturday contributor. Earlier this month, you heard from Soledad Muñiz about  InsightShare’s work with participatory video (PV) for monitoring and evaluation. PV isn’t the only area in which InsightShare is leading the way.

Hot Tip – Photostories: Photostories are a “new format developed by InsightShare for sharing project case studies in an accessible, fun and visually engaging format.” Each photostory is basically a comic that documents a project’s methodology. Here is an example. Click on it to open the high resolution version (it can take a moment to download).

Rad Resource – Example Photostories: More examples from InsightShare may be found online here.

Lessons Learned Using Photostories: I spoke with Soledad via Skype to ask her about InsightShare’s use of photostories. She noted that they offered an engaging option for telling what otherwise can be a dry subject, research methods. They could help to raise the interest of stakeholders and could span linguistic barriers, whether those be due to limited literacy or different languages spoken among stakeholders. Soledad noted that they used the Comic Life software to create their photostories and I decided to give it a try.

Rad Resource – Comic Life: Comic Life is software that helps you to quickly make comics from photos that you have on hand. It is extremely easy to use, with a drag and drop interface and multiple templates. When I demonstrated this in a workshop, an attendee asked if you couldn’t just create comics in another program, perhaps Microsoft’s Publisher or even PowerPoint. Yes! But, Comic Life makes the process considerably easier. The software indexed the files on my hard drive and then I could drag in those I wished to use, add text, and with a couple of clicks, everything was aligned and beautifully laid out and ready to publish. My mind began swimming with the possibilities for using Comic Life in different contexts. After the trial period ended, I happily paid the $29 for the version for my Mac laptop. It is also available for windows, and as a $4.99 ap for iOS devices.

Get Involved – Share your innovative evaluation communication ideas and win an eStudy workshop registration: I’ve written before, encouraging you to think imaginatively about engaging stakeholders around evaluation communication and reporting. Any time before May 10, share your idea for ‘out of the box’ options for evaluation communicating and reporting via the comments on the post. We’ll randomly select one contributor from among all of those making suggestions and you can win a free registration to Kylie Hutchinson’s upcoming eStudy “An Executive Summary is Not Enough: Effective Reporting Techniques for Evaluators” or any AEA eStudy of your choice offered between now and December 31, 2013. New eStudies are added monthly and the current list is online here. While you are welcome to comment multiple times, we’ll only consider one entry per person. You do not need to be an AEA member to enter.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

My name is Soledad Muñiz and I’m the Head of PV M&E at InsightShare. Susan has kindly introduced our work and methodology in this post some weeks ago. And as she anticipated, I’m here today to share some of our experiences using Participatory Video for Monitoring & Evaluation (PV M&E).

Lessons Learned – how we’re using participatory video:

In the last 5 years, we have been developing Participatory Video for Monitoring & Evaluation with a broad range of partners. We’ve seen how Participatory Video allows for organisations to gather qualitative data that often escapes traditional monitoring and evaluation tools. It can monitor the project lifecycle over time and space through interviews, on-site visual monitoring and most significant change stories (MSC).

Hot Tips – integrating participatory video in evaluation:

It is not always easy to gauge and communicate what significance a programme or activity had in the lives of those who intended to help. Those best positioned to explore and convey these messages are those invidividuals – the main actors of development themselves – they can speak first-hand about impacts and outcomes. They can select relevant individuals to interview in their communities and monitor relevant key events as they happen. All actors can reflect back on changes in the community through screenings, where stakeholders are brought together to reflect and discuss.

This video offers a summary of the PV & MSC process in one of our latest initiatives.

A Short Documentary on PVMSC from InsightShare on Vimeo.

Lessons Learned – when and when not to use video:

Our methods help stakeholders tell their stories and communicate their perspectives in an accessible, compelling and versatile format through a participatory and authentic process. Following stringent informed consent procedures, these stories can then be used to communicate lessons or new ideas across to new groups, other organisations or decision makers. It is an overt process, so it’s important to make sure there is an in depth informed consent process through which participants fully understand the implications of sharing their voices in video and can decide on content, the shape of the final product as well as who can watch the video.

Hot Tip – taming the technology:

Experiential learning is at the core of PV M&E. Our motto is “Mistakes are great” and the process is guided by InsightShare’s values & core charter. This encourages participants to feel safe and own the learning space, lose fear of equipment, work at their own rhythm, have fun and enjoy the learning journey. The suite of tools employed include; PV games, editing games, Participatory Learning in Action exercises, visualisation techniques, Theatre of the Oppressed games, role-play and various art exercises.

Rad Resources:

Participatory Video for M&E: in our website you can find plenty of resources about PV M&E, including videos, photostories, case studies and articles.

We’re focusing on video use in evaluation all this week, learning from colleagues using video in different aspects of their practice. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

My name is Susan Kistler and I am the American Evaluation Association’s Executive Director and aea365’s regular Saturday contributor. Are you using video as part of your evaluation work? Over the past couple of weeks, I have had the privilege of speaking with a number of AEA members who are using video in different ways and we are going to develop an aea365 week focusing on video use for evaluation.

Get Involved: Are you using video as part of your evaluation work, whether it be for data collection, analysis, or reporting? If so, add a note to the comments for this post as we’d love to learn about your work!

Lesson Learned – What is Participatory Video? PV brings together a group or community to tell the story of their program or experience. Some AEA members are using PV in evaluation, including colleagues at InsightShare. Here is their explanatory video about participatory video (if viewing this via email, you may need to click back to the aea365 site to see this video).

Lesson Learned – How can PV be used for Evaluation? As one type of PV engagement, InsightShare works with stakeholders to tell the stories of the Most Significant Change in their community. Using the Most Significant Change Technique, evaluators work with stakeholders via a structured process to identify the most significant change – the most significant story of impact – for a program. PV is one strategy for sharing MSC stories.

Rad Resource: During the video week later this year, we’ll hear more from Soleded Muniz at InsightShare – but I encourage you to explore the InsightShare website in the meantime, as they have a wealth of information, examples, and guidance about PV.

Rad Resource – Two Not to be Missed Guides:

Most Significant Change Technique Guide: This free step-by-step guide authored by MSC inventor Rick Davies and his colleague Jess Dart covers everything from how to implement MSC to a more theoretical comparison of MSC to other approaches and epistemologies.

Insights Into Participatory Video Handbook: Available in English, French, Spanish, and Russian, this detailed and beautifully compiled handbook includes guidance for implementing PV, including assistance for facilitators, technical tips, and ethical considerations.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

My name is David Fetterman.  I’m President & CEO of Fetterman & Associates, an international evaluation consulting firm (with 25 years experience at Stanford University) and past-president of the American Evaluation Association (AEA).  I am probably best known for empowerment evaluation work (helping people learn how to evaluate their own programs).  For examples see our blog and an article about empowerment evaluation in the School of Medicine at Stanford University in Academic Medicine, and the book Empowerment Evaluation in the Digital Villages:  Hewlett-Packard’s $15 Million Race Toward Social Justice, Stanford University Press

LGBT-Related Survey

One of my recent evaluations, conducted with my  Stanford School of Medicine students, focused on LGBT curricular training in medical schools throughout the U.S. and Canada.  The results – a median reported time of 5 hours of LGBT-related content in the entire curriculum – were published in this article in the Journal of the American Medical Association (JAMA). It received considerable attention in the press, in part because it is as much a human rights issue as a medical education issue. I’ll share a few tips and tricks that emerged from conducting and publishing this study.

Hot Tip:  We used an online survey program to ask Deans of Schools of Medicine to evaluate their institutions’ level of coverage of 16 LGBT related topics.  Online survey tools, such as SurveyMonkey, save time and money and sort data almost instantaneously.  Surveying Deans automatically enhances the credibility of findings (especially if findings suggest minimal coverage of the material, as in our case).

Reporting survey findings was as much a social responsibility as a scholarly one.  See Anne Dohrenwend’s challenge to speak out about gay rights in Academic Medicine.

Cool Trick: Videoconferencing programs, including Skype, ooVoo, and Google Hangouts are invaluable tools to facilitate communication with team members at remote sites.  Most team members were located across the country, completing residency requirements.  Videoconferencing allowed us to function remotely and inexpensively.

Rad Resource: The Association of American Medical Colleges maintains a curriculum management and Information (CurrMIT) database that helps you determine the coverage of specific topics in medical schools. This database was particularly useful as a form of triangulation when our reporting format – “reported hours of instruction” – was questioned in a draft of our article.

Recommended LGBT cultural competence resources:

Fenway Health

Institute of Medicine of the National Academies

Lesson Learned:  Be prepared for significant opposition to unpopular or controversial findings.  Be prepared to speak with the press.  Highlight key findings and recommendations simply and concisely and be prepared to see how journalists use the information (see example of highlighted findings in New York Times.)  Appreciate your team and enjoy the media blitz for as long as it lasts.

aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. We’re celebrating LGBT Evaluation week with our colleagues in AEA’s LGBT Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. 

blog · cultural competency · lgbt · survey · technology

My name is Emily Spence-Almaguer and I am an Associate Professor of Behavioral and Community Health at the University of North Texas Health Science Center. I spend most of my professional time serving as an independent evaluator for community initiatives and conducting assessment studies. I am a social worker by training and have found that the conversational skills used in Solution-Focused Therapy have great application in the realm of evaluation and community assessment.

Hot Tips: My favorite ways to use solution-focused dialogues are in:

• Focus group and individual interviews because they help generate rich qualitative data and great ideas for continuous program improvements.
• Evaluation planning meetings because they help stakeholders articulate a wide range of potential outcomes and describe how those outcomes might be observed (i.e., measured).
• Meetings where stakeholders are being debriefed around disappointing evaluation results. The nature of solution-focused dialogues avoids finger-pointing and helps drive forward momentum.

Hot Tips:

• It’s all about the questions!! Solution-focused dialogues are driven by questions that promote deep reflection and critical thinking.
• Context: Use questions that help situate people’s minds in a particular context and use details in your question that will encourage an individual to imagine him or herself in that moment. Here’s an example that I use with consumers at a program trying to help lift individuals and families out of poverty:
  • I want you to take a moment and imagine that you just learned that the Bass [local philanthropist] family recently donated $100,000 to the United Way for this project. They want you to help them figure out how to best spend the money. What is the first thing you would advise them to do? What would you advise them to do next?
  • Expertise: I love the way that Gaiswinker and Roessler referred to this as the “expertise of not-knowing”. In solution-focused dialogues the words of questions and tone of delivery are carefully crafted to amplify the assumption that the stakeholders have exceptional knowledge, skills and capacities.

Rad Resource: For an introduction to solution focused concepts, I like Coert Visser’s Doing What Works Blog.

Download from the AEA Public eLibrary to View the Poster in Full Size!

Rad Resource: I presented on Solution-Focused dialogues in evaluation at AEA’s Evaluation 2012 conference. You can download my poster and resources list from the AEA public eLibrary here.

Lessons Learned: A direct question, such as “What would you recommend to improve this program?” often fails to generate detailed or meaningful responses. In focus groups with program consumers, I find that this question is interpreted as “what is wrong with the program?” and may lead to comments in defense of the program staff members (see my 2012 AEA poster for an example of this from my data).

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! My name is Aubrey Perry, the Coordinator for Data Collection and Analysis at the Gateway to College National Network. We work with two-year colleges, school districts, and state departments of education to provide opportunities for high school dropouts and underprepared college students to achieve college credentials.

Hot Tip: As an internal evaluator for a nationwide network, I am partially responsible for communicating our data collection practices and processes to individuals of varying backgrounds and skills. I have found that mind mapping is a great way to graphically show the relationships between various principles and tasks.

Rad Resource: For more information on how you can integrate mind mapping with your evaluation projects, check out Dana Dehart’s post on AEA365.

Hot Tip: When looking for mind mapping software, there are a few important questions you should ask:

1. Is the program flexible enough to accommodate your creative thinking patterns? When most people think of mind maps, the “tree” structure comes to mind where everything branches off a main concept. A good program should do this, and more, by providing the flexibility to design the map that best fits what you’re trying to convey.
2. Will the program connect to the other programs I use in my office or routines? Mind maps are great to include in evaluation proposals or training materials as a way to graphically represent the processes someone might do. To accomplish this, however, you need a way to link the two programs. Make sure the program you choose allows you to save or export the map as a JPG, PNG, BMP, or PDF file.
3. Is the program financially feasible for my practice or organization? There are many great programs out there that can help you start mapping. However, these range from free to very expensive. I recommend starting with one or two free programs to see if they meet all of your needs.

Rad Resource: The program that I use for Mind Mapping is xMind. The java-based software has an extensive set of features available for free users, including all of the features I mentioned above. One of my favorite features is the ability to easily embed images to the tree itself, instantly making the mind map more graphic. You can also share the image to the web, making it easy to embed on your virtual training materials. Here’s a mind map showing some of the processes xMind is great for:

Rad Resource: Sometimes the best inspiration comes from others! Join the xMind user group at biggerplate for a network of xMind users that have posted plenty of examples. Also browse the IconArchive for images that can easily embed into your map.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Mind Mapping

My name is Susan Wolfe and I am the owner of Susan Wolfe and Associates, LLC, a consulting firm that applies Community Psychology principles to strengthening organizations and communities.

I serve as the local evaluator for four programs that are funded by Federal grants for a five year cycle. Each program was required to set objective for each of the five years of funding for a pre-set group of performance indicators. At the end of each year they report their performance objective and their actual performance.

This year I facilitated staff retreats for two of the programs. I compared the objectives and actual performance for each indicator. I tagged the indicators where the program fell short of the objective by a good margin as “red light,” the indicators where the program just missed, or will miss next year as “yellow light,” and the ones where the program met the objective and will do so next year as “green light.” During the staff retreat I reviewed the program’s logic model with staff to show how their activities connected with each indicator, and then we went through the red and yellow light indicators and discussed the challenges associated with meeting the objectives. We then celebrated the success with the green light indicators — which were the largest in number.

For the last two hours of the retreat, I facilitated staff discussions whereby staff developed specific strategies, complete with timelines and clear responsibilities to improve performance for red and green light indicators. Staff at both programs were fully engaged and fully participated in the process, with the end result being a clear plan that was realistic and included accountability.

Lessons Learned: Program staff is interested in learning about the bigger picture of their program, and how their activities relate to performance reporting. If they are empowered with information and an opportunity to develop strategies, they will engage and build their capacity to make their program successful.

Hot Tip: Review logic models with all program staff to show them how their role fits into the larger picture. It helps them to become invested in maintaining accurate data records and in the evaluation process.

Lessons Learned: Staff may need more information to determine which direction to take to improve their performance. Evaluators need to be ready to facilitate that process by gathering or analyzing more data and allowing staff input into the evaluation process

Rad Resource: For more information about Community Psychology, its principles and values and see how they guide our work, see the website for the Society for Community Research and Action.

This is a bonus post from the week sponsored by the American Evaluation Association Community Psychology Topical Interest Group. The contributions from December 9-14 all came from CP TIG members, be sure to return to aea365 and take a look! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org.

SCRA

Hi, my name is Brian Hoessler and I’m a consultant who works with community-based organizations to build their capacity to make change. Previously, I worked in the non-profit sector in Ontario, most recently as a researcher at Kingston Community Health Centres for two projects including a support program for high school students called Pathways to Education. This program was based on a successful social innovation in Toronto’s Regent Park, and as part of the national Pathways initiative we followed their core four-pillar model of support: however, we soon realized that our students in small-town Kingston, and the challenges they faced, were very different from those in multicultural downtown Toronto.

To help our site adapt the national model to local realities, I drew on Developmental Evaluation (DE), a new approach well-suited to projects facing uncertainty and complexity. Instead of assessing adherence to pre-determined plans (formative) or outcomes (summative), DE helps infuse evaluative thinking into the process of program development by framing assumptions and rapidly providing data: in the case of Pathways to Education, I presented findings to the team and helped facilitate conversations around what it meant and how we should respond.

Hot Tips:

• Relationships are key. Building trust is important for any evaluative activity, but more so in DE given the integral role of the evaluator in the team. The evaluator has to feel comfortable in bringing forward feedback (positive and negative) and challenging assumptions; likewise, team members have to trust the evaluator and understand that uncomfortable questions are being raised to help improve the program rather than to pass judgment. Explicitly defining your role is important.

• Build trust by being there. Participate in meetings, help out with programming and events, chat with colleagues during coffee breaks, share interesting finding – demonstrate that you share your colleague’s aim of helping the program to meet its goals.

• Draw on your team members’ front-line experiences. For example, our initial student cohorts were progressing well overall through grades 9 and 10 – findings which didn’t match the experiences at other sites. Speaking with one of our support workers, I learned that there would be a “perfect storm” of pressures and challenges for our students starting in grade 11, an insight I confirmed through historical cohort data from the school boards. Armed with this information, the team could then discuss how to best support those at risk.

Rad Resources:

The McConnell Foundation’s website has two free resources in pdf on DE:

• A Developmental Evaluation Primer
• DE 201: A Practitioner’s Guide to Developmental Evaluation
• A great book, covering theory, methods, and case examples is Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use by Michael Quinn Patton

We’re celebrating all this week with our colleagues in the American Evaluation Association Community Psychology Topical Interest Group. The contributions all week come from CP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org.

developmental evaluation

I’m Tara Gregory, Research and Evaluation Coordinator for Wichita State University’s Center for Community Support and Research (CCSR). CCSR works with non-profit, community and faith-based organizations across Kansas and was originally supported through the Community Psychology graduate program at Wichita State University. Many of our staff members, myself included, are graduates of this program so we’ve maintained a strong community psychology orientation in our principles and practices. Given the principle of meeting people where they are, we often use forms of storytelling to help organizations develop logic models

We use the following techniques to facilitate creative discussion while still attending to the elements in a traditional logic model. These processes encourage participation by multiple staff, administrators and stakeholders and can use the organization’s vision or impact statement as the “happily ever after.”

Hot tip – Script writing: We ask participants to think of their program and it’s outcomes in terms of a movie trilogy. In small groups, they create scripts for each part of the trilogy then report out on the significant scenes (much like they would if they were describing a movie they’d just seen). These scenes inform the elements of their logic model, which we typically help them to complete later, and could be focused on the individual or other contexts (e.g., community). We specifically ask them to think of Part 1 as the story of what people experience while involved in the program; Part 2 picks up at a later date (the specific timeframe depends on the program) and reflects the progression of outcomes; and Part 3 represents the transition to “happily ever after.”

The specific questions we ask participants to address in their scripts are:

• Who are the characters, settings or contexts?

• What do they experience/what happens to them?

• What actions do they take as a result?

Hot tip – Pictorial timeline: Using a similar process to script writing, we ask participants to envision one of their clients, then to draw the activities and resulting behaviors or conditions that occur at various points along a timeline. This approach offers a visual path toward “happily ever after.”

Lessons learned:

• Participants are less likely to get bogged down in concerns about the “right” way to fill out a logic model and are better able to identify outcomes, including those that are unintended or less positive, than with traditional methods.

• Whereas completing the typical logic model matrix can be intimidating for some, these processes tend to be energizing and fun

• These techniques work particularly well with organizations that are innovative and are open to playfulness and experimentation.

We’re celebrating all this week with our colleagues in the American Evaluation Association Community Psychology Topical Interest Group. The contributions all week come from CP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org.

No tags