CAT | Evaluation Policy
We are Dayna Albert (Project Coordinator) and Rochelle Zorzi (Editorial Board Co-chair) of the Evaluation Stories Project, an EvalPartners Innovation Challenge recipient. Our project will soon launch an International Call for Evaluation Stories. The purpose is to:
- Identify and share stories of evaluations that have made a difference
- Increase the demand for and use of evaluation
Minimal literature exists on the benefits or impacts of evaluation use, particularly from the perspective of evaluation users. Furthermore, most evaluation literature is very academic. Our project will employ a story-telling format in order to better communicate the benefits of evaluation use to evaluation users.
As an international project, one of our challenges is to reach a multilingual audience despite limited translation resources. A second challenge is to explain what we mean by evaluation impact – a concept that turns evaluative thinking on its head and tends to be misconstrued.
Lessons Learned: Anticipate that people may have difficulty ‘getting’ a new concept. Words alone can be inadequate and ambiguous.
Use story to explain new concepts. Here is a story that Chris Lysy helped us develop to explain the concept of evaluation impacts.
(Click here to see the video!)
– Follow-up with clients after an evaluation to reflect on and track evaluation impacts.
– Act now! The Call for Evaluation Stories is a great opportunity to reconnect with a client and explore their interest in participating in the Call for Evaluation Stories
– To reach a multilingual online audience, add Google’s Website Translate plug-in to your website. Albeit imperfect, it provides a free and virtually instantaneous website translation.
– To translate a blog, paste the following code into a text widget. Insert your blog’s URL where indicated. The code is written for English (en) to French (fr) translation. For English to Spanish translation, replace ‘fr’ with ‘sp’ and ‘français’ with ‘español’.
<a href=”//translate.google.com/translate?u=http%3A%2F%2FyourblogURL&hl=fr&ie=UTF8&sl=en&tl=fr”” title=””français“”><img src=”http://yourblogURL /2010/02/icons-flag-gb.png” alt=”français” /></a>
Rad Resources: See these posts for additional information on evaluation stories:
- Susan Kistler on VOPEs and the EvalPartners Innovation Challenge
- Chris Lysy on Cartoons and Illustrations as Information Visualization
- Susan Eliot on Evaluation Stories
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
I am Brian Yoder, Director of Assessment, Evaluation and Institutional Research at the American Society for Engineering Education, a professional association located in Washington, D.C. I also serve as President Elect for the Washington Evaluators, a local affiliate of AEA.
I’ve lived and worked in D.C. for the past seven years working as a contractor, in government and a professional society, and I believe government processes can be helped through the use and application of evaluation. As the saying goes, there are no problems, only opportunities, and I’ve seen plenty of opportunities to improve government processes and the improved use of evaluation to assess government programs.
Lessons Learned: Traditionally, I think evaluators have tried to keep their role separate from implementation and the policy-making processes. But, based on my work in D.C., I’ve come to believe that policy makers and program implementers would be well served by evaluators being involved more closely and directly in policy making and program implementation processes. When you work in an environment where the answers to important questions were needed yesterday, and questions that need to be answered keep changing, the traditional approach to evaluation with formative evaluation leading to summative evaluation becomes too slow and irrelevant.
That’s why I helped to spearhead the Evaluators Visit Capitol Hill (EVCH) Initiative, a joint effort between the Washington Evaluators and AEA’s Evaluation Policy Task Force (EPTF). EVCH is an initiative that coordinates attendees at the American Evaluation Association conference in Washington, D.C. this fall to meet with someone in the office of their congressperson so they discuss with them the importance of evaluation and give them EPTF materials.
My hope is that this initiative can accomplish three things:
- Make more policy makers aware of AEA and the work of EPTF.
- Expand the reach of EPTF to creating connections for EPTF.
- Give evaluators the opportunity to be part of the early policy-making process by providing materials on evaluation to policy makers prior to the policy being made.
The deadline to sign up to participate has past, but if you would like to learn more about the initiative, click here http://washingtonevaluators.roundtablelive.org/EVCH
Hot Tip: For those of you participating, please remember to pick up your packet of materials at the Local Affiliates Working Group table located close to AEA conference registration.
Rad Resource: If you would like to know more about the Evaluation Policy Task Force, click here http://www.eval.org/p/cm/ld/fid=129
Rad Resource: If you would like to learn more about the Washington Evaluators, click here http://www.washeval.org/
This is the last of three weeks this year sponsored by our Local Arrangements Working Group (LAWG) for Evaluation 2013, the American Evaluation Association Annual Conference coming up next month in Washington, DC. They’re sharing not only evaluation expertise from in and around our nation’s capital, but also tips for enjoying your time in DC. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to email@example.com.
This post comes to you from Valerie Caracelli, Senior Social Science Analyst at the U.S. Government Accountability Office (GAO). I’m serving with David Bernstein as co-chair of the Local Arrangements Working Group (LAWG) for the 2013 American Evaluation Association Conference in Washington, D.C. October 14-19.
Lessons Learned—Do federal managers use performance information? The conference theme, The State of Evaluation Practice in the Early 21st Century, lends itself to the question of whether and how federal managers use performance information to manage for results. GAO’s most recent periodic survey of Federal Managers was recently released under Managing for Results: 2013 Federal Managers Survey on Organizational Performance and Management Issues, GAO-13-519SP.
Lessons Learned—Do evaluations get used? The conference theme also raises the question of whether and how evaluations can make a difference! Stephanie Shipman spearheaded a companion GAO report that focuses on questions from the Federal Managers Survey that address how evaluations contribute to improving program management and policy making. Through case examples, GAO explored barriers that impede use, and strategies that agencies use to get evaluation results used. See Program Evaluation: Strategies to Facilitate Agencies’ Use of Evaluation in Program Management and Policy Making [GAO-13-570]. One finding is that evaluators rely on a body of evidence, rather than a single study, which allows for responding to a broad array of evaluation questions of interest to decision makers. You can download both reports from www.gao.gov.
[AE1] Hot tips for visiting D.C. in 2013. The perspective you gain from a body of evidence led me to think about your visit to our nation’s capital and where you might find the best perspective for viewing the city and its surroundings:
- Naturally, the Washington Monument comes to mind but that will be closed owing to repairs from the 2011 earthquake!
- At just over half the height of the monument, you can see a 360 degree panorama of the same views from the Clock Tower at the Old Post Office Pavilion. Donald Trump recently purchased the building but the National Park Service still controls the Clock Tower which will remain open during renovations.
- Evenings you can enjoy the view from the W Hotel’s 11th floor and P.O.V. Roof Terrace provide panoramic views of the monuments and D.C. landmarks which are illuminated in the evening.
- Staying awhile? The Washington National Cathedral’s Pilgrim Observation Gallery offers a 360-degree view of the city of Washington and its environs.
For more resources on these and other activities, we encourage each of you to visit the Washington Evaluators Local Affiliate website at www.washeval.org. Washington Evaluators are looking forward to the conference and your arrival!
We’re thinking forward to October and the Evaluation 2013 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Registration is now open! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to firstname.lastname@example.org.
1 Comment · Posted by Susan Kistler in Evaluation Managers and Supervisors, Evaluation Policy, Government Evaluation, Organizational Learning and Evaluation Capacity Building
I am Andy Blum, Vice President for Program Management and Evaluation at the U.S. Institute of Peace (USIP), an independent organization that helps communities around the world, prevent, manage or recover from violent conflict. I recently spoke at a brown bag for the Washington Evaluators about the process of improving my organization’s learning and evaluation processes in general, and creating an organizational evaluation policy in particular. In this post, I’ll provide a few of the takeaways that could be applicable in other contexts.
Here are three, hopefully generalizable, lessons from the process of crafting the evaluation policy at USIP.
Lesson Learned: Conducting a baseline assessment was extremely helpful. By asking staff their greatest hopes and fears of evaluation, themes emerged, and the findings proved useful as an assessment of where we stood regarding learning and evaluation, and for developing an action plan to improve evaluation.
Lesson Learned: When talking about changing how evaluations are done and used in organizations, you need to manage messaging and communications almost fanatically. The phrase “demystify evaluation” had real resonance. I found myself becoming almost folksy when discussing evaluation. Instead of saying theory of change, I asked, “Why do you think this is going to work?” Instead of saying indicator or metric, I would ask, “What are you watching to see if the program is going well?” Especially at the beginning of an effort to improve evaluation, you do not want to alienate staff through the use of technical language.
Lesson Learned: There is a tension between supporting your evaluation champions and creating organizational “standards.” Your evaluation champions have likely have created effective boutique solutions to their evaluation challenges. These can be undermined as you try to standardize processes throughout the organization. To the extent possible, standardization should build on existing solutions.
Rad Resource: The best change management book I’ve seen: Switch: How to Change Thing When Change is Hard, by Chip Heath and Dan Heath.
Hot Tips—Insider’s advice for Evaluation 2013 in DC: The Passenger is DC’s most famous cocktail spot, but if the weather is good you can’t beat Room 11 in Columbia Heights as a place to sit outside and drink real cocktails.
We’re thinking forward to October and the Evaluation 2013 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Registration is now open! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.
My name is Susan Kistler and I am the American Evaluation Association’s Executive Director Emeritus and aea365’s regular Saturday contributor. Today, we’re talking about VOPEs and a unique opportunity for global impact.
Lesson Learned – VOPEs: What’s a VOPE? A VOPE is a Voluntary Organization for Professional Evaluation (VOPE). According to the freely downloadable Evaluation and Civil Society: Stakeholders’ Perspectives on National Evaluation Capacity Development, “VOPEs include formally constituted associations or societies, as well as informal networks and communities of practice. Their memberships are open not only to those who conduct evaluations but also to those who commission and utilize evaluations and those engaged in building the evaluation field.”
Rad Resource – EvalPartners: I’ve written before about EvalPartners, their great training and background resources, and their work to map and strengthen collaboration among VOPEs. This month, EvalPartners has announced a new project to strengthen the field of evaluation around the world through thinking outside the box.
Hot Tip – EvalPartners Innovation Challenge: The EvalPartners Innovation Challenge aims to “identify [and implement] innovative ideas to strengthening the demand for and use of evaluation to inform policy making, including in the context and spirit of EvalYear, the international year of evaluation. Ideas can relate to proposed actions at international, national and/or sub-national levels.” It is a small grants program, providing US$15,000 to each of three winning proposals. Eligible candidates are VOPEs, and partnerships including VOPEs, and innovation is defined broadly.
Get Involved: Be sure to read all of the details at http://www.mymande.org/evalpartners/innovation_challenge. What ideas do you have to increase the demand for and use of evaluation in the policy making arena? Share via comments or discuss with your evaluation colleagues.
I am Melanie Hwalek, CEO of SPEC Associates and a member of AEA’s Cultural Competence Statement Dissemination Core Workgroup. My focus within the Workgroup is to help identify ways to disseminate the Statement and integrate its contents into evaluation policy. AEA’s Think Tank: Adoption of the AEA Public Statement on Cultural Competence in Evaluation: Moving From Policy to Practice and Practice to Policy gave me three big ideas for doing this.
Lesson Learned: Cultural Competence can be in big “P” policy and small “p” policy. Dissemination of the Cultural Competency Statement doesn’t have to start with federal or state level, big “P” policy change. Small polices like setting criteria for acceptable evaluation plans, for assuring that evaluation methods take culture into consideration, and for ensuring culturally sensitive evaluation products can go just as far – or further – in assuring that all evaluations validate the importance of culture in their design, analysis, interpretation and reporting.
Hot Tip: Start where there is a path of least resistance.Agencies that exist to represent or protect minority interests are, themselves, culturally sensitive. These are the agencies that should easily understand the importance of assuring that the evaluations of their programs should include cultural competence. If you are passionate about infusing cultural competence into municipal, state or federal policy, start with these types of agencies since they are likely to understand the importance of culturally sensitive evaluations. Keep in mind, though, that just because an organization “says” it values cultural competence doesn’t mean the really know how to be and act in a culturally competent way.
Hot Tip: Try to go viral.Infusing cultural competence into policy means that we need to be open to all kinds and levels of policy, much of which is identified only through practice. The lesson here is to start promoting cultural competence to anyone and anywhere evaluation planning, methods, analysis and reporting are discussed. In this networked world, the more people who think and talk about cultural competence in evaluation, the more likely it will find its way into evaluation practice and evaluation policy.
Rad resource: William Trochim wrote an informative article on evaluation policy and practice.
This week, we’re diving into issues of Cultural Competence in Evaluation with AEA’s Statement on Cultural Competence in Evaluation Dissemination Working Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to email@example.com. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
My name is Robert McCowen and I am a doctoral fellow in Western Michigan University’s Interdisciplinary Ph.D. in Evaluation. I served as a session scribe at Evaluation 2010, and attended session number 651, Introduction to Evaluation and Public Policy. My evaluation interests focus on education, and a great deal of modern educational policy flows from the top down—so it only makes sense to find out as much as possible about how policy is made, and how evaluators can make sure their voices are heard.
Lessons Learned: George Grob, the presenter, has a long history of involvement with evaluation and government. Among his many past positions is a 15-year term as the Director of the Inspector General’s Office of Evaluation and Inspections. He had a number of wise statements for evaluators:
- “Home runs” do happen in government, but that’s not how games are won. Rejoice if your work finds instrumental use in legislation or regulation, but don’t make it your only goal.
- Get to know the gatekeepers in government, whether at the federal and state level. Work with them, listen to them, keep them informed, be willing to respect their schedules, and you’ll have a much easier time making sure your reports get to where they can do the most good.
- Know the relevant body of work when you deal with policymakers. Assume they know everything important about the topics they deal with (because they might), and strive to do the same.
- When writing reports, you have maybe two pages to catch the eye and make a case for your conclusions. Make sure your best evidence and most compelling findings are obvious to readers.
- Be as professional as possible, including making sure your integrity and independence are unimpeachable—but be careful to keep lines of communication and cooperation open with major policymakers and other stakeholders.
Great Resource: Mr. Grob’s presentation is an excellent resource for any evaluator who is new to dealing with government, and can be found here at the AEA public eLibrary.
At AEA’s 2010 Annual Conference, session scribes took notes at over 30 sessions and we’ll be sharing their work throughout the winter on aea365. This week’s scribing posts were done by the students in Western Michigan University’s Interdisciplinary PhD program. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.
Hi, my name is Michelle Baron. I am the Associate Director of The Evaluators’ Institute, an evaluation training organization, and the chair of the curating team for aea365.
As a retired Army veteran, I have conducted many evaluations with a wide range of stakeholder support. I have found three techniques to facilitate a well-received evaluation:
Cool Trick #1: Cultivating an environment for teaching and learning helps to put organizations at ease when going through the evaluation process. When you take away the “I gotcha!” and replace it with valuable instruction organizations can use for future improvement, you help to build a bridge of trust between you as the evaluator and the organization. When organizations contact YOU with evaluation ideas for their workplace, you know a good working relationship is blossoming.
Cool Trick #2: Referring organizations to helpful resources (both online and offline) helps to increase their self-sufficiency and foster productive conversations before, during, and after the evaluation. Military websites often have links to regulations and manuals that foster development of criteria and standards for a given topic.
Cool Trick #3: Increasing evaluation capacity by offering evaluation training in a given area (e.g., physical fitness, vehicle licensing) helps the organization to become not only familiar with policies and procedures of a particular content area, but helps them to be proactive and to think evaluatively regardless of whether they’re being formally evaluated.
I hope this Veteran’s Day brings you more in tune with the needs of your military stakeholders and that you can approach evaluation with a caring and helpful attitude so stakeholders will see the value in the work and reciprocate accordingly.