AEA365 | A Tip-a-Day by and for Evaluators

TAG | capacity

Hi there! We’re Anjie Rosga, Director, and Natalie Blackmur, Communications Coordinator, at Informing Change, a strategic consulting firm dedicated to increasing effectiveness and impact in the nonprofit and philanthropic sectors.  In working with clients large and small, we’ve found that organizations are in a better position to learn if they take the time to prepare and build their capacity to evaluate. To facilitate this process, Informing Change developed the Evaluation Capacity Diagnostic Tool  to measure an organization’s readiness to take on evaluation.

Rad Resource: The extent to which evaluation translates into continuous learning is in large part dependent on the organizational culture and level of evaluation experience among staff. These are the two primary categories—themselves divided into six smaller areas of capacity—in the Evaluation Capacity Diagnostic Tool. The tool is a self-assessment survey that organizations can use on their own, in preparation for working with an external evaluator or alongside an external evaluator. A lower score indicates that an organization should, for example, focus on developing outcomes and indicators, track a few key measures or develop simple data collection forms to use over time. The higher the score, the higher the evaluation capacity; staff may then be able to collect more types and a greater volume of data, design more sophisticated assessments, as well as integrate and commit to making changes based on lessons learned.cap tool graphic

However, there’s more to the Evaluation Capacity Diagnostic Tool than the summary score. It is a powerful way to catalyze a collective discussion and raise awareness about evaluation. Taking stock and sharing individuals’ perceptions of their organization’s capacity can jumpstart the process of building a culture that’s ready to evaluate and implement learnings.

Hot Tip: Make sure everyone is on the same page. Especially if an organization is inexperienced in evaluation, it’s important to discuss the vocabulary in the Tool and how it compares with individuals’ own definitions.

Hot Tip: Assessing evaluation capacity can be a tough sell. Organizations come to us because they’ve made the decision to begin evaluation, but gauging their capacity to do so can feel like a setback. To get organizations on board, we frame evaluation capacity as an investment in building a learning culture and the infrastructure that can make the most of even relatively limited data collection efforts.

We love to hear from folks who have implemented or reviewed the tool! Feel free to reach out to us at news@informingchange.com.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Cindy Banyai of the Refocus Institute, a consultancy specializing in participatory evaluation using visual methods, and graduate of Ritsumeikan Asia Pacific University, here to discuss my experience working and researching in Asia. This post was inspired by the AEA365 post from Efrain Gutierrez on interviews with Latinos. Efrain’s perspective on cross-cultural data gathering resonated with me and I wanted to share how I think it relates to social research and evaluation in Asia.

Lesson Learned – Relationships matter and it takes time to build them. The Chinese call it guanxi, but I found it to be important all over Asia. Don’t expect to be able to get a straight or in-depth answer from someone you just met. Use informal interviewing techniques and snowball sampling after being introduced to key informants through a trusted acquaintance. Allow ample time for relationship building for in-depth interviews and recognize the potential shallowness of one-off interviews and surveys.

Lesson Learned – There is a strong deference to authority in Asia. This is particularly true if the country has a Communist government or history. In my experience, this leads interview respondents to the answers they think you are looking for, wreaking havoc on data gathered from multiple choice surveys. People may be uncomfortable and take a long time with open-ended questions, but the data will be less biased by this factor.

Lesson Learned – Asia has a love/hate relationship with the West. I’ve experienced exuberance, resentment, curiosity and downright hostility to my work and presence in Asia. In developing countries I was often seen as someone who could provide aid, causing interviewees to paint an overly sunny picture of the situation. While in Japan, Westerners are treated respectfully but find themselves perpetually outside of the system, making it extremely difficult to peer through the veneer. The best advice I have in this regard is to understand the context in which you are working and consider it in your approaches to data-gathering and analysis.

Rad Resource – This is most definitely not an exhaustive list of the lessons I’ve learned during my 8 year Asian tenure, but they are some basic constructs relevant to gathering quality data in social research and evaluation. If you’re interested in reading about my experience in the Philippines, where I employed extensive relationship-building and informal interviewing techniques, check out my book Community Capacity and Development.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Archives

To top