Hello, we are Sheila Matano and Dani O’Neill from Carson Research Consulting (CRC). We at CRC are currently in the process of conducting a survey on relocation and housing stability. Although surveys are an incredibly useful method for gathering quantitative information from large populations, response rates vary widely. In order to promote our project, interact with the community and hopefully increase our response overall rate, CRC conducted a Community Survey Day for residents in addition to providing the survey online and mailing out paper surveys.

Lesson Learned: Planning and preparation.
Planning is essential! Staffers started planning for community survey day about two months in advance.  We had meetings to discuss logistics and created a master task list to make sure everything we needed would be ready for the day of the event. Google Drive provided a way to easily edit and update important documents that could be accessed by all team members.

Lesson Learned: Know your target audience!
Using background information on the target audience, we attempted to tailor the event to the resident’s needs. We found a central location that was easily accessible by public transportation and also offered free parking. In addition, we gave residents the option of taking the survey online, completing a paper survey or taking the survey over the phone. We wanted to make sure that respondents could choose an option that was most comfortable for them.

Lesson Learned: Advertise!
We advertised the event in three ways:

• We sent a letter to our targeted residents informing them of the event
• We created event flyers that were distributed to residents
• We also made phone calls to residents a few days before the event

Hot Tip: The Day of the event
Community Survey Day was hosted on a Saturday to give residents who work during the week the opportunity to attend, and we also offered breakfast and lunch to all attendees.  Staff interviewers were available to do survey interviews with residents from 9am to 5pm. The interviewers read questions aloud to the residents and inputted their responses directly into SurveyMonkey via laptops; residents did not use computers themselves.

Hot Tip: Having the survey online made it readily accessible to interviewers and made the process of completing surveys faster and easier than using paper surveys (although we did have paper surveys on hand in case technology failed us!).

Outcome: Overall, Community Survey Day was a success. We had the opportunity to directly interact with residents and get valuable feedback about our survey. Residents appreciated that they could come in and talk with us directly, as well as offer their suggestions about our project.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

survey

My name is Soledad Muñiz and I’m the Head of PV M&E at InsightShare. Susan has kindly introduced our work and methodology in this post some weeks ago. And as she anticipated, I’m here today to share some of our experiences using Participatory Video for Monitoring & Evaluation (PV M&E).

Lessons Learned – how we’re using participatory video:

In the last 5 years, we have been developing Participatory Video for Monitoring & Evaluation with a broad range of partners. We’ve seen how Participatory Video allows for organisations to gather qualitative data that often escapes traditional monitoring and evaluation tools. It can monitor the project lifecycle over time and space through interviews, on-site visual monitoring and most significant change stories (MSC).

Hot Tips – integrating participatory video in evaluation:

It is not always easy to gauge and communicate what significance a programme or activity had in the lives of those who intended to help. Those best positioned to explore and convey these messages are those invidividuals – the main actors of development themselves – they can speak first-hand about impacts and outcomes. They can select relevant individuals to interview in their communities and monitor relevant key events as they happen. All actors can reflect back on changes in the community through screenings, where stakeholders are brought together to reflect and discuss.

This video offers a summary of the PV & MSC process in one of our latest initiatives.

A Short Documentary on PVMSC from InsightShare on Vimeo.

Lessons Learned – when and when not to use video:

Our methods help stakeholders tell their stories and communicate their perspectives in an accessible, compelling and versatile format through a participatory and authentic process. Following stringent informed consent procedures, these stories can then be used to communicate lessons or new ideas across to new groups, other organisations or decision makers. It is an overt process, so it’s important to make sure there is an in depth informed consent process through which participants fully understand the implications of sharing their voices in video and can decide on content, the shape of the final product as well as who can watch the video.

Hot Tip – taming the technology:

Experiential learning is at the core of PV M&E. Our motto is “Mistakes are great” and the process is guided by InsightShare’s values & core charter. This encourages participants to feel safe and own the learning space, lose fear of equipment, work at their own rhythm, have fun and enjoy the learning journey. The suite of tools employed include; PV games, editing games, Participatory Learning in Action exercises, visualisation techniques, Theatre of the Oppressed games, role-play and various art exercises.

Rad Resources:

Participatory Video for M&E: in our website you can find plenty of resources about PV M&E, including videos, photostories, case studies and articles.

We’re focusing on video use in evaluation all this week, learning from colleagues using video in different aspects of their practice. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hi there. I’m Corey Newhouse, the Founder and Principal of Public Profit, based in Oakland, California. We help public service organizations measure and manage what matters.

Lessons Learned – how we’re using video: Public Profit uses video in our evaluation of the Partnerships for Learning (PFL) initiative of the National Equity Project.

We use video as an elicitation technique in our teacher interviews. That’s a fancy way to say that we take video of PFL teachers in the fall, and then show the footage again to them during a follow-up interview in the spring. When teachers can see themselves in the classroom, they are able to be very specific about the ways in which their practice has changed while receiving coaching from PFL, and helps outsiders (including us!) better understand the changes they describe.

We work with a professional videographer to get high quality footage, and usually edit the video in-house. We have also taken our own video using inexpensive handheld cameras. If you chose to shoot your own footage, invest in a lapel microphone, as it makes a big difference in the quality of the footage.

Hot Tips – integrating video in evaluation: Enhancing subjects’ ability to recall their performance improves the quality of our interview data, particularly when we’re interested in such a complex phenomena as teaching.

Plus, we have video of teachers’ practice to share, improving the explanatory value of our reports.  Speaking of, here’s a clip of teachers’ practice, along with their reflections on their work with students. If you can’t view the video below in your browser or email, you can view it directly at: http://www.youtube.com/watch?v=Huw9IYY8hO4.

Lessons Learned – when and when not to use video: Take time to negotiate with your client – and their clients, if needed – about when videotaping will take place, and how it will be used. This will help to put your subjects at ease and assure that the shoot day runs more smoothly. A one-page description of your video project is a really useful conversation starter.

Hot Tip – taming the technology: It pays to plan when taking video! Collaborate with your team (including a videographer, if you have one) on a moment-to-moment calendar for your shoot day, including what kind of footage will be taken and when. While shooting, make notes to yourself about when particularly important moments were taped so that you can find the footage easily during editing.

Rad Resources: Video in Qualitative Research by Heath, Hindmarsh, and Luff, is an exceptionally helpful guide to those of us new to using video in evaluation.

We’re focusing on video use in evaluation all this week, learning from colleagues using video in different aspects of their practice. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

My name is William Rickards; I am currently senior research associate in the Office of Program Accreditation and Evaluation at USC Rossier School of Education. My career has largely been focused in higher education, although I have worked in program evaluation in delinquency prevention, youth services, and in a range of educational and social services.

Lessons Learned-How I’m using video: Over the last few years I have been particularly interested in the use of video-recording for interviews; in my case, this has usually meant interview studies with students and graduates. Use is primarily as data collection, although I use select segments for reporting to faculty; I do the taping on my own, often with portable equipment.

Two examples:

• In evaluating the use of an undergraduate learning e-portfolio, I interviewed graduates regarding their use of the portfolio to monitor and assess their own development
• In an evaluation for a graduate teacher education program for teachers in international schools, I interviewed the teachers on their paths into international school work to understand how to best meet their needs

Hot Tips—Considerations when using video in evaluation include:

• The video as a particularly rich artifact presents potential challenges in terms of analysis: How will the transcript be handled? How much depth will be included in the text?
• At another level, the video record offers a unique opportunity—and often a stark one—from which to study and hone one’s own skills as an interviewer.
• Additionally, the video artifact can provide material that can be used in reporting, depending on clearances, in presentations, websites, or project videos.

Hot Tips—Taming the technology

• The biggest consideration with the technology (particularly in field settings) will be the microphone. External mics—that plug into the camera—are usually best, even if they must often be purchased separately.
• Data storage and transfer need to be studied in relation to individual situations, equipment, and comfort levels.
• Power will always be a consideration—as in battery life and access to a power supply.

Rad Resources

The ethics of informed consent and participation are always a concern, but video complicates this because of participant identity recorded visually. For example, it is standard practice to de-identify data that are being stored for analysis, but this is difficult with video records. These factors need to be considered in the consent and video release forms.

• The American Anthropology Association’s statement on Ethnography and Institutional Review Boards may be helpful in this regard.
• The AEA Guiding Principles are also helpful.
• There is considerable information available on YouTube and Vimeo under ethnographic interviewing that can be useful for evaluation applications (e.g., http://vimeo.com/1269848 ).

We’re focusing on video use in evaluation all this week, learning from colleagues using video in different aspects of their practice. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello, we are Linda Cabral and Laura Sefton from the Center for Health Policy and Research at UMass Medical School. We often collect qualitative data from interviews and focus groups. One challenge we frequently face is how to quickly and efficiently transcribe audio data. We have experimented using voice recognition software (VRS), and we’d like to share our approach.

You will need headphones, a microphone (stand-alone or attached to a headset), and a computer with audio playback and VRS installed on it. We use Dragon Naturally Speaking Premium Version 11.5 voice recognition software, however other VRS is available. Use of audio playback software will allow you to control the playback speed, so you can slow it down, pause, fast forward, and rewind as needed.

Open the audio file in the playback software and open a new document in the VRS. While listening to the audio via the headphones, repeat what you hear into the microphone. During this step, you can format the document to indicate who is speaking and to add punctuation. Because VRS works best when trained to understand a single voice, a designated team member should repeat all spoken content, regardless of how many voices are in the audio file.

This process will generate a document in the VRS that can be saved to your computer as a Word file. As a final review, read through the Word file while listening to the audio file and make needed corrections. This could be done by another member of the project team as a double check of the document’s accuracy.

Hot Tips:

• Spend time training the VRS to recognize your voice. A few practice sessions with the software may be needed where you can read dummy data into the software in order for it to learn your voice. This will improve the transcription quality, minimizing the time spent editing.
• Train the VRS to recognize project-specific acronyms or terminology prior to starting transcription.

Lessons Learned:

• Often, financial resources for evaluation projects are limited. In an effort to keep the transcription process in-house, our administrative staff transcribed the audio files. By using the VRS and someone from our project team familiar with the data as the designated recorder, we have found savings in time and efficiencies.
• No transcription yet has captured 100% content accurately the first time. Therefore, build in time to listen to the recording and to make manual edits.

Rad Resources:

These resources may be helpful as you explore whether VRS is right for you.

• “The Voice Transcription Technique: Use of Voice Recognition Software to Transcribe Digital Interview Data in Qualitative Research” by Jennifer Matheson

• VRS products Review by consumersearch: “In reviews, it’s generally Dragon vs. Dragon”
• 

  (Share Clip)

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

qualitative · technology · transcription

Hi, we are Christine Johnson and Terri Anderson, members of the Massachusetts Patient Centered Medical Home Initiative (MA-PCMHI). MA-PCMHI is Massachusetts’ state-wide, multi-site PCMH demonstration project engaging 46 primary care practices in organizational transformation to adopt the PCMH primary care model.  Our roles as Transformation and Quality Improvement Director (Christine) and Qualitative Evaluation Study Team Lead (Terri) require us to understand the 46 practices’ progress towards PCMH model adoption in distinct yet complementary ways.  Our colleagues sometimes assume that we must remain distant to conduct our best possible work.  Their concerns are that our close working relationship will somehow contaminate the initiative or weaken the evaluation’s credibility.  However, we find that maintaining our connection is vital for success of both of the initiative and the evaluation.  We’d like to share the following:

Lessons Learned:

• Transformation and Quality Improvement (Transformation/QI) and evaluation both seek to understand how the practices best adopt the PCMH model and to describe the practices’ progress.  To promote our mutual interest, we regularly attend each other’s team meetings. Doing so increases the opportunity to share our perspectives on the MA-PCMHI. To date the evaluators have advised some formative project adjustments while the MA-PCMHI intervention team has increased the evaluators’ understanding of the survey and performance data submitted from the practices. Currently, the project team and the evaluators collectively are establishing criteria to select six practices for in-depth site visits.
• Transformation/QI and evaluation often use the same data sources but in different ways.  Specifically, the practices use patient record data in their Plan-Do-Study-Act (PDSAs) cycles then submit the same data for the evaluation’s clinical impact measures.  The practices initially resisted this dual data use.  However, through our Transformation/QI-Evaluator connection we increased the practices’ understanding of how their use of data in the PDSAs improved their clinical performance which in turn improved the evaluation’s ability to report a clinical quality impact. Presently, performance data reporting for clinical impact measures and practices’ use of PDSAs have increased.

Hot Tip: Develop a handout describing the similarities and differences between research, evaluation and quality improvement.  Having this information readily available has helped us to address concerns about bias in the evaluation.

Rad Resources:

• To learn more about Transformation/QI see the Agency for Healthcare Research and Quality’s 2010 Consensus Report on Practice Facilitation for Primary Care improvement.
• To learn more about PDSAs see the Institute for Healthcare Improvement’s Plan-Do-Study-Act tools.

(Share Clip)

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Capacity Building · qualitative

Hello! We are Johanna Morariu, Kat Athanasiades, and Ann Emery from Innovation Network. For 20 years, Innovation Network has helped nonprofits and foundations evaluate and learn from their work.

In 2010, Innovation Network set out to answer a question that was previously unaddressed in the evaluation field—what is the state of nonprofit evaluation practice and capacity?—and initiated the first iteration of the State of Evaluation project. In 2012 we launched the second installment of the State of Evaluation project. A total of 546 representatives of 501(c)3 nonprofit organizations nationwide responded to our 2012 survey.

Lessons Learned–So what’s the state of evaluation among nonprofits? Here are the top ten highlights from our research:

1. 90% of nonprofits evaluated some part of their work in the past year. However, only 28% of nonprofits exhibit what we feel are promising capacities and behaviors to meaningfully engage in evaluation.

2. The use of qualitative practices (e.g. case studies, focus groups, and interviews—used by fewer than 50% of organizations) has increased, though quantitative practices (e.g. compiling statistics, feedback forms, and internal tracking forms—used by more than 50% of organizations) still reign supreme.

3. 18% of nonprofits had a full-time employee dedicated to evaluation.

4. Organizations were positive about working with external evaluators: 69% rated the experience as excellent or good.

5. 100% of organizations that engaged in evaluation used their findings.

6. Large and small organizations faced different barriers to evaluation: 28% of large organizations named “funders asking you to report on the wrong data” as a barrier, compared to 12% overall.

7. 82% of nonprofits believe that discussing evaluation results with funders is useful.

8. 10% of nonprofits felt that you don’t need evaluation to know that your organization’s approach is working.

9. Evaluation is a low priority among nonprofits: it was ranked second to last in a list of 10 priorities, only coming ahead of research.

10. Among both funders and nonprofits, the primary audience of evaluation results is internal: for nonprofits, it is the CEO/ED/management, and for funders, it is the Board of Directors.

Rad Resource—The State of Evaluation 2010 and 2012 reports are available online at for your reading pleasure.

Rad Resource—What are evaluators saying about the State of Evaluation 2012 data? Look no further! You can see examples here by Matt Forti and Tom Kelly.

Rad Resource—Measuring evaluation in the social sector: Check out the Center for Effective Philanthropy’s 2012 Room for Improvement and New Philanthropy Capital’s 2012 Making an Impact.

Hot Tip—Want to discuss the State of Evaluation? Leave a comment below, or tweet us (@InnoNet_Eval) using #SOE2012!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Capacity Building · Data · Evaluation · nonprofit · organizational learning · reports

Welcome to the Evaluation 2013 Conference Local Arrangements Working Group (LAWG) week on aea365. I’m Megan Walker from Communities In Schools.  When I think about how I’ve grown in my life, I see a cycle.  I start out with a goal or a need (wanting to lose weight); I work with supportive people who can help me achieve that goal (healthy friends); I look back occasionally to see what’s been working or not working (exercise is helping, Primanti’s is not); I adjust what I’m doing (more salads); and I start the cycle over again (back on the treadmill).

That’s what led me to where I am today – my favorite part of the country, D.C.  That’s what led me to being married to a man I love, and that’s what led me to a fulfilling job at Communities In Schools, which uses a similar model to ensure that students stay in school.

Lesson Learned – Take Advantage of all that Data

Not coincidentally, that’s the same kind of practice that I bring to my work on the Research, Evaluation, and Innovation team here.  When I first started, we were in the early stages of taking our network of almost 200 affiliates through an accreditation process called the Total Quality System (TQS).  The goal is to ensure that affiliates are aligned with our model.  We evaluate this by reviewing documentation of business and operational practices and conducting interviews with affiliate stakeholders.

As our team went through this process, we realized that we were conducting about 160 interviews per semester, but using the data only minimally.  We decided to adjust, using a tool that I had used previously as a researcher at the University of Pittsburgh – NVivo.

Rad Resources – Tools for Qualitative Data Collection and Analysis

NVivo is an excellent tool for gathering, coding, and analyzing data.  I am currently completing the second round of our Community Stakeholder Study, which I conduct yearly by analyzing themes and trends in the hundreds of interviews that we collect during accreditation.  I highly recommend consulting resources such as Pat Bazeley’s book on qualitative data analysis – and the people at the help desk at QSR International have also been useful.

I have just two other useful tidbits to think about when embarking on a qualitative journey.  First, consult your experts.  The people who prepare our affiliates have been invaluable in giving feedback on our interview questions.  Second, be pithy.  People who work with at-risk students want quick and pertinent information.  With that in mind, I crafted several different targeted versions of the report, which will be used across our network to improve practice.

Hot Tip—Insider’s advice for Evaluation 2013 in DC: Join me for a conference run around the monuments this October 2013! Contact me at meganreneewalker@gmail.com for more details.

We’re thinking forward to October and the Evaluation 2013 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). AEA is accepting proposals to present at Evaluation 2013 through until March 15 via the conference website. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

No tags

Friends and colleagues, I am Enid Schmidt and I have been conducting qualitative and quantitative research for over twenty years both as a consultant and as an internal evaluator. Ironically, I find that there are relatively few qualitative online resources, at least when compared to the myriad quantitative offerings. Here, I share three of my favorites and invite you to add your own via the comments section of this post on aea365.

Rad Resource – the Qualitative Report: The qualitative report is a “weekly online journal dedicated to qualitative research since 1990.” This open access online journal offers well researched articles that span the gamut of qualitative inquiry, including an extensive host of reviews of current qualitative texts. Bogged down by an awful search function, I recommend using the article index to get a feel for the journal’s scope.

Clipped from: www.nova.edu (share this clip)

Rad Resource – the Listening Resource: I was introduced to Susan Eliot’s blog when she wrote about it on aea365 in December of 2011, back when it was called the Qualitative Research Blog. This is my “go-to” resource for ideas that I can apply immediately in my own work. You can tell that Susan is conducting qualitative research in her day to day practice and she offers grounded advice on how to improve the down and dirty doing of qualitative inquiry. Don’t miss the very useful set of free downloadable guides on everything from “Information Rich Sampling” to “Recording Focus Groups” that can be found on her Free Downloads page.

Rad Resource – Research Design Review: The counterpoint to the Listening Resource, RDR proclaims that “if it is not already obvious, this is not a how-to blog.” Authored by Margaret Roller, it asks the question “is it good research” and digs deep into research design of all types. Here, there is thoughtful examination of qualitative research, never giving it short-shrift to its quantitative cousin. Roller examines everything from bias and transparency to measurement error and meaning finding, and does so both articulately and succinctly.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello, we are Greg Lestikow, CEO and Fatima Frank, Project Manager of evalû, a small consulting firm that focuses exclusively on rigorous evaluations of social and economic development initiatives.  We champion impact evaluation that maintains academic rigor but is based entirely on our clients’ need to improve strategic and operational effectiveness and increase profitability.

In a recent project, we were tasked with designing a qualitative instrument to complement quantitative data around the sensitive topic of gender-based violence.

Rad Resource: We approached this challenge by designing a focus group discussion (FGD) protocol informed by an article on the “Participatory Ranking Method” (PRM), in which participants rank potential indicators from most to least important. PRM acknowledges project beneficiaries as experts and recognizes the local community as capable of identifying and measuring their progress towards positive change. As such, PRM incorporates local perspectives in the construction of research instruments. By using PRM, we were able to select indicators that are meaningful to the project’s local beneficiaries (in our case adolescent girls affected by violence) and reflective of the concepts they find useful when tracking their own progress. PRM is an ideal evaluation methodology for measuring awareness of sensitive topics and tracking outcomes over time, particularly for projects that may not see any kind of impact in the short or medium term.

Hot Tips:

• Start with a participatory activity to gauge local perspectives and to understand which social practices are considered more or less acceptable in the community. In our case, we asked participants what gender-based violence meant to them.
• To facilitate ranking, show a series of cards labeled with different kinds of social practices (in our case: Shout; Insult, Threaten, Push, Hit, Beat, Kill) and have participants order them from the most to the least acceptable, asking them to explain their decisions.  Alternatively, participants can free-list social practices that are common in their communities and then rank-order them.
• Include an open-ended discussion to understand which social practices are acceptable in different relational and social contexts.

Lessons Learned:

• Make sure moderator and note-taker are gender appropriate.
• If you want to obtain a broad range of perspectives but anticipate potential problems with mixing certain community members in the same FGD, create a few FGD groups and separate participants.
• Ask local evaluation or project teams about any other cultural practices to consider before an FGD. For example, in Sierra Leone we started each FGD with a prayer, as this is a standard practice when people meet.

Please share your stories on challenges, solutions, and experiences in dealing with sensitive topics by leaving a comment here or contacting us.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

participatory evaluation