AEA365 | A Tip-a-Day by and for Evaluators

TAG | MESI

This is Jean King, professor of Evaluation Studies at the University of Minnesota and mother of the Minnesota Evaluation Studies Institute (MESI—pronounced “messy” because evaluation is that way). MESI began 20 years ago to provide high quality evaluation training to all comers: evaluation practitioners, students, accidental evaluators, and program staff and administrators. We are fortunate to have had Minnesotans Michael Quinn Patton and Dick Krueger as regular MESI trainers from the beginning and, with funding from Professor Emerita Mary Corcoran, guest sessions from many of our field’s luminaries. Over the years MESI has taught me a great deal. This entry details three learnings.

Lesson Learned: Structured reflection is helpful during evaluation training. Experiential educators remind us that merely having an experience does not necessarily lead to change; reflection is the key to taking that experience and learning from it. At MESI plenaries we regularly build in time when the speaker finishes for people to “turn to a neighbor” (groups of 2 to 4–no larger) and talk about what they took as the main ideas and any confusions/questions they have. The reflection is easy to structure, and people engage actively. If appropriate, the facilitator can ask people to jot down their questions, which can become the basis of Q&A.

Hot Tip: I never ask an entire large group, “Are there any questions?” At the end of sessions in large conferences/training sessions, the facilitator/presenter will frequently ask the entire group if there are any questions. In these situations there is often an awkward pause, sometimes lasting long enough that people start glancing nervously at each other or at the door, and then someone who can’t stand the silence thinks of a question, raises a hand, and is instantly called on. Everyone breathes a sigh of relief. When I facilitate a session, I instead use the “turn to a neighbor” strategy (briefly—just a couple of minutes) so that everyone can start talking and generate potential questions. You can even call on people and ask what they were discussing in their small group.

Cool Trick: Create Top Ten lists as part of a meeting or training session. Since MESI’s inception, attendees have participated in an annual tongue-in-cheek Top Ten competition where they submit creative answers to a simile that describes how evaluation is like something else (e.g., the state fair, baseball, Obamacare). We provide prizes for the top three responses, and I am continually impressed with people’s cleverness. This year’s topic compared evaluation to interstellar space travel, and the final list is posted at www.evaluation.umn.edu. The Top Ten is a useful activity because it spurs creativity and helps a group come together around a common, low-key cause.

The American Evaluation Association is celebrating MESI Spring Training Week. The contributions all this week to aea365 come from evaluators who presented at or attended the Minnesota Evaluation Studies Institute Spring Training. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

My name is Donna M. Mertens and I am an independent consultant based in Washington DC; my work is both domestic and international. I had the honor of being the keynote speaker at the Minnesota Evaluation Studies Institute (MESI) in March 2015. The MESI theme was Social Justice amidst Standards and Accountability: The Challenge for Evaluation. The concept of social justice in the context of evaluation implies that evaluators can play a role in addressing those wicked problems that persist in society, such as violence, lack of access to quality education for all, poverty, substance abuse, and environmental pollution.

Lesson Learned: Wicked problems and Social Justice. Evaluators are concerned and involved in contributing to the solution of wicked problems. They also recognize the importance of bringing a social justice lens to this work. Michael Harnar conducted a survey of 1,187 evaluators and reported that 69% (n=819) either strongly or somewhat agreed with this statement: Evaluation should focus on bringing about social justice.

Rad Resource: Mertens, D.M. editorial: Mixed Methods and Wicked Problems, Journal of Mixed Methods Research, 2015, 9, 3-6. Abstract http://mmr.sagepub.com/content/9/1/3.extract

Harnar, M. (2014). Developing criteria to identify transformative participatory evaluators. JMDE. http://journals.sfu.ca/jmde/index.php/jmde_1/article/view/383

Lesson Learned: Social Justice Lens Leads to Different Evaluation Questions. Evaluators who work with a social justice lens are concerned with the question of program effectiveness and answering the impact question, Did “it” work? They are also interested in asking other types of questions:

  • Was “it” the right thing?
  • Was “it” chosen and/or developed and implemented in culturally responsive ways?
  • Were contextual issues of culture, race/ethnicity, gender, disability, deafness, religion, language, immigrant or refugee status, age or other dimensions of diversity used as a basis for discrimination and oppression addressed?
  • How were issues of power addressed?
  • Do we want to continue to spend money on things that don’t work?

Rad Resource: Native American Center for Excellence published Steps for Conducting Research and Evaluation in Native Communities that provides a specific context in which a social justice lens is applied in evaluation.

Lessons Learned: Social Justice Criteria for Evaluators. Evaluators who work with a social justice lens consider the following criteria to be indicators of the quality of the evaluation:

  • Emphasizes human rights and social justice
  • Analyses asymmetric power relations
  • Advocates culturally competent relations between the evaluator and community members
  • Employs culturally appropriate mixed methods tied to social action
  • Applies critical theory, queer theory, disability and deafness rights theories, feminist theory, critical race theory, and/or postcolonial and indigenous theories

Rad Resource: Reyes J., Kelcey J., Diaz Varela A. (2014). Transformative resilience guide: Gender, violence and educationWashington, DC: World Bank.

The American Evaluation Association is celebrating MESI Spring Training Week. The contributions all this week to aea365 come from evaluators who presented at or attended the Minnesota Evaluation Studies Institute Spring Training. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

My name is Melissa (Chapman) Haynes from the University of Minnesota’s Minnesota Evaluation Studies Institute (MESI). At MESI we have a strong interest in building evaluation capacity within the university and the community through university-community partnerships. We are trying to build this capacity in a sustainable manner, and in a way that builds upon the practice of professional evaluators and creates scholarship of the teaching of and professionalization of program evaluation. One of our signature activities is a spring evaluation training that MESI has hosted for the past 20 years. This week of posts will highlight a bit of the key learning, resources, and tools presented at our 2015 event!

Lesson Learned: Creating an inclusive community of evaluators is essential but we are an incredibly diverse field – what brings us together? Through the week of MESI the Program Evaluation Standards (Yarbrough et al., 2010) and AEA Guiding Principles were utilized in various contexts. In particular, as a frame of reference as we decide which evaluation projects we will engage in, as a guide to navigation and negotiation of situations where ethics are in question, and to elevate the profession of evaluation in various contexts. We can and should continue to use and explore how these guiding documents can further the professionalization of our field.

Hot Tip: Donna Mertens provided some wonderful examples of the art and power of questioning during her workshop and keynote address. During her workshop she gave some examples of how she uses questioning to negotiate with clients. For example, if a potential client asked you to frame an evaluation in a manner that did not jive with the Program Evaluation Standards or AEA Guiding Principles, one might tell the client something like “I will not do X, but let’s talk about how we might frame an evaluation that will continue to serve the population of interest.”

Rad Resource: Some of the presenters have opted to share the information they presented on our website: http://www.cehd.umn.edu/OLPD/MESI/spring/2015/default.html

Rad Resource: A fun highlight of MESI is the annual “Top Ten” competition. For those new to MESI, Jean King develops a Top Ten statement – this year it was “How is program evaluation like interstellar space travel?” We had over 50 entries from MESI attendees – the Top Ten is located here. My favorite is #2 – “You’ve got to remember that YOU are the alien here.”

The American Evaluation Association is celebrating MESI Spring Training Week. The contributions all this week to aea365 come from evaluators who presented at or attended the Minnesota Evaluation Studies Institute Spring Training. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Archives

To top