AEA365 | A Tip-a-Day by and for Evaluators

TAG | dialogue

Greetings! My name is Shana Alford and I am an internal program evaluator at ACCESS Community Health Network, a 501 (c) federally qualified health center network of more than 40 health centers in under-served communities within the City of Chicago. As an internal evaluator, I objectively analyze how federally funded community health programs are implemented in targeted communities and determine measurable impact and outcomes.

In my experience, I have learned the art of asking the right questions to promote positive dialogue and build relationships. Let’s face it, sometimes you can be viewed as an outsider, even when you are an insider!  Asking the right questions doesn’t imply there is a set of wrong questions, but questions can stimulate dialogue and serve as a powerful medium for conversation, so it is important to set the stage to gather the information you need. Keep in mind that one or more conversations can lead to positive change or at least increase awareness and learning among staff, management, and you too!

The Art of Asking the Right Evaluation Questions

photo credit: WingedWolf via photopin cc

Hot Tips:Asking questions for evaluation purposes is an art and here are three that I use frequently:

1) Insight: Listen to your program team and learn about their unique experiences, their responsibilities, successes and challenges. When you have insight about a program then you are more likely to probe deeper into areas that seem to raise a red flag, or deserve attention because they are going so well. Staff will find pleasure that you know the ins and outs of their program.

2) Relevance: An effective question will be right for the moment and relevant for the group of people you are addressing. I have learned the hard way that asking questions to management that should be asked to program staff and vice versa can cause an awkward case of silence, frustration, or sometimes misunderstanding. It is important to know your audience.

3) Patience: Asking questions should not feel like an interrogation to staff or management. Also, they may not be able to answer a question or feel uncomfortable for many reasons. Therefore, it is importance to practice patience. If the purpose of the question is to learn something new, highlight an existing issue, or clarify, then the evaluator should give the program team time to respond, even if it is at a later date. Hint: If people are unresponsive to a question, sometimes taking a step back and asking the same question, but differently will yield the results you are looking for. This may sound unlikely, but it is true, try it!

Rad Resources:

The American Evaluation Association is celebrating the Chicagoland (CEA) Evaluation Association Affiliate Week with our colleagues in the CEA AEA Affiliate. The contributions all this week to aea365 come from our CEA members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.


· ·

Hi, my name is Bikash Kumar Koirala. I work as a Monitoring and Evaluation Officer in the NGO Equal Access Nepal, which is based in Kathmandu, Nepal.  I have been practicing monitoring and evaluation work for over five years, which is focused on development communication programs.  A research project that EAN has collaborated on Assessing Communication for Social Change (AC4SC) developed a participatory M&E toolkit based on our experiences.  One of the modules in this toolkit is the Communication Module, which is summarized as follows.

As a result of AC4SC, the communication systems in our organization improved a lot and became more participatory. We began to understand that effective communication and continuous feedback is essential to the success of participatory M&E. Communication inside organizations and outside can be quite challenging sometimes because different people have different perspectives and experiences.

Lessons Learned

Community Involvement: After the AC4SC project, the level of engagement with communities by the M&E team increased considerably. Their involvement in ongoing participatory research activities and providing critical feedback has proved very useful to our radio program development. This has increased community ownership of our programs. As well as work undertaken by the M&E team, this research is conducted by network of embedded community researchers (CRs).  These activities have produced research data, which is analyzed and triangulated with the other sources of data (such as listeners’ letters) to produce more rigorous results.

Internal Communication: Regular constructive feedback related to program impact and improvement is given to content teams by the M&E team.  This has increased dialogue and cooperation between the M&E and content team members.  Before the AC4SC project, content team members didn’t usually take M&E findings into account because they felt that they already knew the value of the program content through positive feedback from listener letters. The value of M&E has now been recognized by the content teams. They now ask for more in-depth data to generalize feedback they receive. The M&E team addresses this through research and analysis using many different forms of data from varied sources.

Use of New Communication Technology: The M&E team has been analyzing SMS polls, text messages, and letter responses, and triangulating these with the CRs research data and short questionnaire responses to present more rigorous results to program team members, donors and other stakeholders.

Some Challenges: In participatory M&E it is important to understand the roles of everyone involved in the process. Effectively presenting results for better communication and the utilization of M&E findings among different stakeholders is an ongoing challenge. Time to effectively undertake participatory M&E and is also an ongoing challenge.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · · · · · ·

My name is Daniel Brandão and I am an evaluator from Brazil. I work at Instituto Fonte as a consultant and we have a participatory approach to evaluation. I used to work with evaluation of programs focused in youngsters under social exclusion situations where methods oriented by arts were very useful. In 2007 to face the challenge of creating a good environment to have in-depth conversations with youngsters in conflict with the law we developed the FRAMES Method. Inspired by comics, FRAMES is a set of 27 drawings used to trigger dialogues with these youngsters.

Rad resource: You can get the complete set of drawings at: http://tinyurl.com/ygqafsk. We didn’t produce any article or so about FRAMES. But it is in our 2010 to-do list! A good resource about arts based research practice is the book “When Method Meets Art” (Levy, P. Guilford Press)

FRAMES

  • Can be adjusted for multiple possibilities of applications.
  • Can be used in evaluations or as an educational tool.
  • Can be applied Individually or with groups.
  • With both genders
  • With youngsters over 12 year old
  • Used by psychologists, social service assistants, non formal educators, social scientists and others.
  • Application time lasted from 20 minutes to 5 hours.

The linked document is written in Portugese, but here’s a link to an online translation program: http://www.freetranslation.com/

Regards!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

· · · ·

Archives

To top