AEA365 | A Tip-a-Day by and for Evaluators

TAG | stakeholders

Hi, I’m Pat Christian and founder of Caleb Missionary Relief Services, an international nonprofit that evolved from evaluating children needs with vulnerabilities in Haiti.  My organization has implemented interventions to improve their quality of education and has made a difference with thousands of Haitian students.  I’ve conducted project level evaluations with input from stakeholders for decision making and accountability. Domestically, I’ve served as a seasonal grant reviewer for the Georgia Department of Education and evaluated educational programs from schools and nonprofits applying for federal funds.

I recently attended the 2016 Summer Evaluation Institute in Atlanta. At one workshop, a registrant from Africa inquired how to engage stakeholders in planning a program evaluation. In his country, he’d been asked to evaluate the success of the program at its end without prior involvement. Working internationally, I understood his concern and knew of situations where the overseer of the program did not engage the community and the program was a failure.

Rad Resource: “It’s not the Plan, It’s the Planning: Strategies for Evaluation Plans and Planning” a workshop by Dr. Sheila Robinson at the 2016 Summer Evaluation Institute incorporated a participatory approach for engaging stakeholders in a comprehensive evaluation plan. Dr. Robinson did an awesome job expounding upon five steps for planning a program evaluation while engaging stakeholders in the entire process. Based on her steps, I’ve suggested how international stakeholders can be involved in each step to maximize the evaluation.

Hot Tip: Engage stakeholders. Identify stakeholders who should be involved in planning a program evaluation and develop a plan how to engage them. Stakeholders can provide information on why the program evaluation is needed for the community and can be the program staff, community members and leaders, the participants, collaborative NGO partners, the nationals’ government, others from similar programs, etc.

Hot Tip: Focus the evaluation. Stakeholders can give input for the logic model and share ways how they will use the evaluation. Stakeholders, as an advisory group, can be a strong asset with getting the community to understand the significance of the evaluation, getting information disseminated more widely, and getting more participants to respond.

Hot Tip: Collect data. To ensure cultural competence in evaluation, stakeholders give the evaluator an understanding of the cultural dynamics and can recommend what data collection methods are best for the culture.

Hot Tip: Analyze & interpret the information. As an American, I may interpret data through one set of lens but involving stakeholders to discuss the analysis and lessons learned presents a different lens for interpreting data.

Hot Tip: Use the Information. Discuss with stakeholders how the information should be best communicated with the community and determine the next steps.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

My name is Bonnie Richards, an analyst from Foresee and Chair of the Organizational Learning and Evaluation Capacity Building TIG. Welcome to the OL-ECB sponsored AEA365 week!

This week our blog posts will cover a range of experiences discussing challenges and successes we have had sustaining learning or evaluation in our work with organizations or programs. Across our members’ varied experiences, you will learn more about their strategies and methods for facilitating learning and the challenges they have encountered.

In my own role working with clients, one of my main goals is to help them understand where to prioritize improvements for their stakeholders. One of the challenges in doing this is navigating the different environments of organizations, companies, and government agencies. Each group is unique. For example, among government agencies, while there are some similar requirements or processes that consistently govern each, the mix of involved stakeholders who serve as the primary point of contact actually vary significantly.

A primary contact could be a program analyst, or a director of the agency’s strategic planning and evaluation office, or technical director, or even a third party vendor.

Understanding and acclimating to each client, meeting them at their “level” and working within their context is key because it helps you to learn the best ways for interacting with different stakeholder groups. This sets the stage for a successful relationship.

Lessons learned: Ask questions.

  • So, how does one get to the point of successfully meeting stakeholders in the appropriate context? Ask questions:
  • Why are they beginning this process? Were they instrumental in initiating it, or are they tasked it as part of a directive from a director or committee? How do they intend to use the information? What are their goals? What information will be most useful?
  • Take some time to ask questions. Stakeholders will appreciate your interest and the opportunity, and it exposes you to the thoughts, concerns, and values that are top of mind to the people you will be working closely with.

The American Evaluation Association is celebrating Organizational Learning and Evaluation Capacity Building (OL-ECB) Topical Interest Group Week. The contributions all this week to aea365 come from our OL-ECB TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello, we are Emily Lauer and Courtney Dutra from the University of Massachusetts Medical School’s Center for Developmental Disability Evaluation and Research (CDDER). We have designed and conducted a number of evaluations of programs and projects for elders and people with disabilities. In this post, we focus on the topic of person-centered evaluations. We have found this type of evaluation to be one of the most effective strategies for evaluating aging and/or disability services, as it tends to provide results that are more valid and useful through empowering consumers in the evaluation process.

Why person-centered evaluation? Traditional evaluations tend to use a one-size-fits-all approach that risks supplanting judgment about consumers’ individual perspectives and may not evaluate components that consumers feel are relevant. In a person-centered evaluation, consumers of the program’s or project’s services are involved throughout the evaluation process. A person-centered evaluation ensures the program or project is evaluated in a way that:

  • is meaningful to consumers;
  • is flexible enough to incorporate varied perspectives; and
  • results in findings that are understandable to and shared with consumers.

Lessons Learned:

Key steps to designing a person-centered evaluation?

  1. Design the evaluation with consumers. Involve consumers in the development process for the evaluation and its tools.
  2. Design evaluations that empower consumers
    • Utilize evaluation tools that support consumers in thinking critically and constructively about their experiences and the program under evaluation. Consider using a conversational format to solicit experiential information.
    • Minimize the use of close-ended questions that force responses into categories. Instead, consider methods such as semi-structured interviews that include open-ended questions which enable consumers to provide feedback about what is relevant to them.
    • Consider the evaluation from the consumer’s perspective. Design evaluation tools that support varied communication levels, are culturally relevant, and consider the cognitive level (e.g. intellectual disabilities, dementia) of consumers.
  1. Involve consumers as evaluators. Consider training consumers to help conduct the evaluation (e.g. interviewers).
  2. Use a supportive environment. In a supportive environment, consumers are more likely to feel they can express themselves without repercussion, their input is valued, and their voices are respected, resulting in more meaningful feedback.

Hot Tip: Conduct the evaluation interview in a location that is comfortable and familiar for the consumer. When involving family or support staff to help the consumer communicate or feel comfortable, ensure they do not speak “for” the consumer, and that the consumer chooses their involvement.

  1. Involve consumers in synthesizing results. Involve consumers in formulating the results of the evaluation.

Rad Resource: Use Plain Language to write questions and summarize findings that are understandable to consumers.

Many strategies exist to elicit feedback from consumers who do not communicate verbally. Use these methods to include the perspective of these consumers.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

I’m Alice Walters, a member of AEA’s Graduate Student and New Evaluator TIG.  I am a doctoral student in human services and work as a non-profit consultant in fund development, marketing, and evaluation.  Here, I explore potential pitfalls and recommendations based on experience with stakeholders for new evaluators.

Hot Tip 1:  Stakeholders are central to evaluation – include them in every step of the process.

This may be Evaluation 101 but it bears emphasizing.  Identify, include, and inform stakeholders.  Think carefully and critically about all involved parties in evaluation outcomes.  Leaving out key stakeholders may lead to poor quality evaluation in unrepresented perspectives.  Key decision-making stakeholders should be engaged in the evaluation process to ensure evaluation relevancy. 

Rad Resource: Engaging Stakeholders  This CDC guide has a worksheet for identifying and including stakeholders in evaluation.

Hot Tip 2:  Be proactive in frequent & ongoing communication to stakeholders.

Don’t assume that initial evaluation conversations and perspectives haven’t changed without your knowledge.  Frequent communication with stakeholders will alert you to any changes in stakeholder perspectives toward the evaluation.  Ongoing communication will also keep lines of communication open and inform stakeholders of evaluation progress.

Rad Resource: A Practical Guide for Engaging Stakeholders in Developing Evaluation QuestionsThis 48-page resource from the Robert Wood Johnson Foundation covers engaging stakeholders throughout the evaluation process.  It provides worksheets and a range of useful communication strategies.

Hot Tip 3:  Take the time to consider stakeholder’s views at every stage of evaluation.

Stakeholders may be unclear about the evaluation process, its steps, and methods used.  Be sure to explain and continue to inform at every stage of evaluation.  As a new evaluator, I made the faulty assumption that stakeholder views were unchanging from initial evaluation meetings.  I also failed to use opportunities to communicate during evaluation stages that might have signaled changing circumstances from stakeholder response.  Evaluators should be cautious about assuming that evaluation environments and stakeholder views are static.

Rad Resource: Who Wants to Know? A 4-page tip sheet from Wilder Research on stakeholder involvement. Evaluators have an expertise that may require working away from direct stakeholder contact, particularly key decision-making stakeholders.  The relevancy of an evaluation requires ongoing stakeholder input.  Successful evaluation requires keeping communication channels open with stakeholders.

AEA is celebrating GSNE Week with our colleagues in the Graduate Student and New Evaluators AEA Topical Interest Group. The contributions all this week to aea365 come from our GSNE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi, I’m Susan Eriksson, a geologist and science educator reformed as an evaluator for science-related programs.  I write from my experience as a scientist turned evaluator with many years of working with evaluators, doing my own internal evaluation, and now doing evaluation for others.

Lesson Learned: It seems that many people really ‘don’t get’ evaluation.  “Why do we need this?”  “Evaluators just make work for themselves.” “You put WHAT in the budget!” Grants administrators, financial people, boards and advisory committees, heads of organizations, and STEM Principal Investigators commonly ask why evaluation is important and why it costs so much.

As an independent evaluator, I am still educating people about what evaluation is.  One of the more interesting comments I’ve heard recently was from a program officer in an un-named federal agency.  “Susan, why would anyone hire YOU?  Evaluators are social science researchers!” Although a reformed scientist/educator  does not necessarily qualify as an evaluator, many people equate social science research with evaluation.

Evaluation is deemed increasingly important by our government – knowledge- generation faster and supported by evidence!  People giving out the grants want to do the ‘right thing’ but many admit they don’t know what good evaluation looks like.  In addition, many grant proposal reviewers are inexperienced in evaluation.  I just sat on a review panel in which the relatively inexperienced science faculty spoke highly of proposals who mention the phrase ‘external evaluator’.  At Evaluation 2013, an NSF officer told us to always include a logic model because reviewers are just beginning to understand those.

We have a long way to go for people to understand the breadth and depth of good evaluation.

Hot Tip: Continue to use any opportunity to educate your clients, your peers, your friendly grant administrator about what evaluation is, what good evaluation looks like, and why evaluation is important in helping people ask the right questions and get significant answers.

Rad Resources: Three websites are great for our colleagues and clients who need a boost in evaluation:

  1. Better Evaluation is an international collaboration to improve evaluation practice and theory by sharing information about options and approaches.
  2. National Science Foundation; the well-used 2002 User-Friendly Handbook for Project Evaluation
  3.  And a tip from my colleague Ayesha Tillman writing in this same STEM evaluation blog series, Read and become familiar with AEA’s Guiding principles for evaluators.”
Clipped from http://betterevaluation.org/

 

Susan Eriksson is a leader in the newly formed STEM Education and Training TIG. Check out our TIG Website for more resources and information.

The American Evaluation Association is celebrating STEM Education and Training TIG Week with our colleagues in the STEM Education and Training Topical Interest Group. The contributions all this week to aea365 come from our STEM TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

My name is Mike Morris and I’m Professor of Psychology at the University of New Haven, where I direct the Master’s Program in Community Psychology. My research focuses on ethical issues in evaluation, and I am an Associate Editor of the American Journal of Evaluation. The best book I’ve ever read for managing my relationships with stakeholders in an evaluation was not written by an evaluator, nor was it written specifically for evaluators.

Rad Resource: Peter Block (2000). Flawless Consulting: A Guide to Getting Your Expertise Used (2nd ed.). San Francisco: Jossey-Bass. http://www.josseybass.com/WileyCDA/WileyTitle/productCd-0787948039.html

2014 Update:  Flawless Consulting is now in its 3rd edition (2011).

Among organizational consultants this book is legendary. Evaluation is, in my view, one form of consultation, so it’s not surprising that Block’s book is relevant to our work. His discussion of such issues as entry/contracting, dealing with resistance, and managing the feedback of results is invaluable. Central to his analysis is the concept of “authenticity,” which means putting into words what you are experiencing with stakeholders as you work with them. It might sound a bit scary at first, but the more you practice it, the more effective at managing these relationships you become. I also believe that Block’s approach to consulting can enhance the ethical quality of evaluations, especially in terms of helping evaluators identify strategies for raising and pursuing ethical issues with stakeholders.

Flawless Consulting is exceedingly well-written. It probably helps that Block does not have a doctoral degree, since writing a dissertation is a process that can extinguish one’s ability to compose a sentence that anyone would be interested in reading. Flawless Consulting gets very positive reviews from my students. I hope you’ll agree with them. 

The American Evaluation Association is celebrating Best of aea365 week. The contributions all this week are reposts of great aea365 blogs from our earlier years. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

My name is Derrick Gervin and I currently work as a Lead Evaluator at The Evaluation Group (TEG) in Atlanta, Georgia. I work with school systems and nonprofit organizations to improve student achievement. After completing my first six months as a full-time evaluator, I would like to share some tips with other newcomers to evaluation.

Hot Tips:

  • Look inside: Identify your strengths and how they may be used in the evaluation. I realized early on that the field of evaluation was too diverse for me to know everything so I chose to pull from my strengths as a social work practitioner.
  • Relationship building: The more you know about your client and their work, the better you can meet their evaluation needs. I’ve taken to doing a Google search of both the client’s organization and key people in the organization – going beyond just their website helped me to uncover valuable information to assist in my work. Also, I take advantage of opportunities to interact with clients during special events (i.e., career fairs, book festivals, and trainings).
  • Build trust and be accessible: Make commitments and keep them. Ask clients for their input. Set aside time to be available to clients and return calls and emails as soon as possible. I have monthly evaluation meetings to discuss successes and challenges. Also, I spend as much time as possible on site meeting with project staff and observing processes.
  • Get Organized: Find an organizing system that works for you. Also, plan to do as much project management as direct evaluation work. Especially, when projects are at the beginning stage. I’ve found a need to take continuing education classes in time management and the use of Microsoft Excel. I’m constantly searching for ways to maximize my time and work more efficiently.
  • Conceptualization: Explore techniques to assist in conceptualizing planned work and expected outcomes. I regularly visit AEA365 for helpful data visualization tips and conceptualization resources. I really like DoView for creating logic models.
  • Professional Development: Take advantage of opportunities to increase evaluation knowledge and skills. Know your limitations and consult with mentors and other evaluators in the field. I’ve found my co-workers to be a great source for answering and/or talking through challenging evaluation related issues. I participate in monthly lunch and learn sessions, as well as, group conference calls where we discuss and receive feedback on our evaluation projects.

The American Evaluation Association is celebrating Best of aea365 week. The contributions all this week are reposts of great aea365 blogs from our earlier years. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

Hello! I am Lara Hilton MPH, and I am a research analyst at RAND Corporation (Santa Monica, CA), Samueli Institute (Alexandria, VA), and a doctoral student at Claremont Graduate University (Claremont, CA). I have worked in evaluation contexts such as hospitals, clinics, military settings, and international development with a myriad of stakeholders. I have a few guidelines that I find helpful when developing new relationships. Akin to a first date, the first meeting with stakeholders is usually rife with anxiety, expectations, and assumptions. I hope these suggestions help you navigate the waters of this delicate period of relationship building:

Hot Tip: Be informed. Read ahead by accessing as much information as possible about the organization, people, program, mission, goals, and setting. The more you know about them, the more intelligent your inquiry will be. This suggestion cannot be overstated.

Hot Tip: Be alert. Have your antennae out to get a quick read on stakeholders. In a first meeting, many people reveal opinions, expectations, orientations, values, prejudices. Pay special attention to their level of evaluation expertise, interest in evaluation, and goals for what they want to get out of the evaluation. Conflicting opinions may begin to arise that are best negotiated early in the relationship.

Hot Tip: Be flexible. Match your language to theirs. If they are research or academically inclined then use evaluation jargon. If they do not have evaluation expertise then it is best not to overwhelm with meaningless terms. If they use special terms like acronyms then adopt them immediately. Get on the same page with language as soon as possible to enhance communication.

Hot Tip: Be of service. The most effective way to overcome the negative reputation of our profession is to be of service to the organization, transparently. State it up front. I have literally heard a sigh of relief from program directors and staff when they hear me say that I am here to serve them. Part of this process is to make sure that research questions include not only “does it work” but also “how” and “why” so that negative final results are accompanied by their context and feedback loops for improvement may be provided, if appropriate.

Rad Resource: The CDC’s first step in the program evaluation framework is to engage stakeholders. A practical guide for determining how and to what extent to involve stakeholders in evaluation can be found here.

Rad Resource: Donaldson’s 2001 article: Overcoming our negative reputation: Evaluation becomes known as a helping profession (American Journal of Evaluation, 22, 355-361.)

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello, I’m Jori Hall, assistant professor at the University of Georgia and a member of the AEA Public Statement on Cultural Competence in Evaluation Dissemination Working Group. This tip is focused on integrating cultural competence into everyday practice through values-engagement.

Tips:

  • As suggested in the AEA Public Statement on Cultural Competence in Evaluation, all evaluation practice and evaluands are situated in and influenced by cultural norms, values, and various ways of knowing. Values-engagement acknowledges these influences and attempts to be responsive to the dynamic interaction between the values reflected in evaluation practice and the evaluand. That is, values-engaged evaluators understand that evaluation practice promotes values, and that these values must respectfully engage stakeholders’ values.
  • Values-engagement is not a specific strategy or a set of required methods; rather, it is a commitment to culturally responsive evaluation. While there is more than one way to be values-engaged, the commitment to culturally responsive, values-engagement suggested here involves the evaluator prioritizing values of inclusion and equity in everyday practice. Inclusion refers to engaging and describing the plurality of stakeholders’ values, perspectives, and concerns, focusing on the least well served in a particular context. Equity refers to how well and to what extent the evaluand is attending to stakeholder groups (i.e., access, participation, etc.) in the context. Because values-engagement advocates inclusiveness and the equitable treatment of stakeholders, it supports the goals of the Statement on Cultural Competence in Evaluation.
  • Values-engagement can be integrated throughout the life cycle of an evaluation, and enacted through generating evaluation questions, data, and dialogues related to the ways in which the evaluand is attending to the cultural values of the groups represented in the context. To learn more about values-engagement, its connection to cultural competence, and how evaluators can practically enact its commitments in different evaluation contexts, begin with the resources provided below!

Rad resources:

The American Evaluation Association will be celebrating Cultural Competence Week. The contributions all this week come from the Cultural Competence committee. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

· · · · · · · ·

Hi, I’m Aimee White and I am a Strategic Learning and Evaluation Consultant at FSG in the Seattle office and I have been a practicing evaluator for nearly 10 years.

Lessons learned – In my world, evaluation is only good if it is informative for the client and produces actionable learning. This means having a real partnership with the clients, engaging them as often as possible in the evaluation processes. However, involving “stakeholders” in the evaluation process can be challenging. Deciding who should be involved and at what stage creates complexity in evaluation planning and the more stakeholders included, the more challenging the work becomes! Now imagine working with society’s most challenging populations, those whose voices are often undervalued and the challenge just grew.  Even though it might be challenging, I believe that  authentic and meaningful engagement of key stakeholders in the design of the evaluation questions, and subsequent processes, makes for a more meaningful, relevant, and actionable evaluation overall. It is worth the effort to include the most relevant voices in your projects.

Rad Resource – I want to share with you today a tool “A Practical Guide for Engaging Stakeholders in Developing Evaluation Questions”, created by Hallie Preskill and Nathalie Jones of FSG, during a project with the Robert Wood Johnson Foundation, that can be used by evaluators to determine when it is appropriate and how best to engage stakeholders in the design of the evaluation questions. The guide is a step by step manual on how best to engage stakeholders and includes checklists and many practical tools.

engaging stakeholders

I would like to draw your attention to the Stakeholder Engagement Strategies and Criteria matrix on p. 23. Using this chart can help evaluators determine if she/he needs to plan focus groups, one on one meetings with key stakeholders, or if surveys will be sufficient to gather meaningful input in the design of the evaluation questions. Once you have established the criteria and strategies that best suit your stakeholder group a variety of techniques are then presented that will create great learning for the evaluator as well as for the stakeholders. The guide also has worksheets that will improve your practice of better engaging stakeholders.

matrix

The guide is extremely useful, practical, and easy to use. I highly encourage you all to consider how better to engage your stakeholders in the work you are doing today.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Older posts >>

Archives

To top