AEA365 | A Tip-a-Day by and for Evaluators

TAG | performance measurement

We are Wanda Casillas and Heather Evanson, and we are part of Deloitte Consulting LLP’s Program Evaluation Center of Excellence (PE CoE). Many of our team members and colleagues are privileged to work with a variety of federal agencies on program evaluation and performance measurement and, throughout this week, will share some of their lessons learned and ideas about potential opportunities to help federal agencies expand the value of evaluations.

This week members of our team will share lessons learned about working remotely on federal evaluations, the use of qualitative methods in federal programs that don’t always appreciate the value of mixed methods, the potential for federal programs to be more “selfish” in program planning, the value of conducting evaluation and performance measurement for federal programs, and making the most out of data commonly collected in federal programs. In the coming weeks, readers will find an additional article on scaling up federal evaluations.

Lesson Learned: Many federal clients use performance measurement, monitoring, evaluation, assessment, and other similar terms interchangeably; however, evaluators and clients don’t always have the same definitions, and therefore expectations, in mind for what these terms mean. It’s important to learn as much as possible about your federal client’s experiences and history with evaluation through research and conversations with relevant stakeholders in order to make sure you can deliver on a given agency’s needs.

Lesson Learned: Clients sometimes see evaluation or performance measurement as a requirement rather than an opportunity to understand how to improve upon or expand an existing program. As evaluation consultants, we sometimes have to work with clients to help them understand how evaluation can benefit them even after responding to a request for proposals.

Rad Resource: Alfred Ho provides some intriguing insights on the effects of the Government Performance and Results Act of 1993, which has resulted in many of the performance measurement and evaluation activity we see today in GPRA after a Decade: Lessons from the Government Performance and Results Act and Related Federal Reforms.

The American Evaluation Association is Deloitte Consulting LLP’s Program Evaluation Center of Excellence (PE CoE) week. The contributions all this week to aea365 come from PE CoE team members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hi, we’re Eva Bazant, evaluation staff at Jhpiego, and Vandana Tripathi, consultant to global public health programs. Jhpiego is an affiliate of Johns Hopkins University in Baltimore, Maryland, working to improve maternal, newborn and child health globally.

In many sectors, such as education, observations are used for professional evaluation. We are sharing lessons learned from an experience of using structured observation in evaluation of health care quality offered on the day of birth in low-resource settings (our experience was in Madagascar’s hospitals), carried out by the Maternal and Child Health Integrated Program of USAID.

Lessons Learned:

  1. Build trust of the individuals being observed and the professionals in charge to allow the observation to happen. Rely on a respected senior colleague to negotiate entry for observers.  Communicate clearly how the data will be used and kept secure, and to whom findings will be disseminated.
  2. Build in enough time to train and standardize observers’ competencies in observation; this can help identify potential challenges with the observation process and tools. Train observers to be a “fly on the wall” and stay long enough to allow employees to feel at ease and act normally, thereby reducing the Hawthorne effect.
  3. Use the shortest checklist/tool needed to cover important topics, to reduce error and fatigue; Validate the tool with topical experts prior to use, and pretest in the field.
  4. Create clear response categories to minimize ambiguity and need for interpretation by observers. Clarify for observers the distinctions between “not observed”, “not done” and “not applicable.”
  5. Interview your observers at the end, and communicate frequently during the process, to document how the observer tools were used. Review cases for completeness and discuss missing data.
  6. Use technology when possible (e.g., smart phone data entry) to increase efficiency in data entry. Ensure observers are comfortable using and maintaining the technology.
  7. Triangulate data from multiple sources to affirm and contextualize observation findings. Observation findings can be compared with interview or inventory data.

Lesson Learned Highlight – Improve validity and inter-rater reliability: During observer training, carry out one or more exercises to promote consistency of data. Have a trainer perform a complex service, omitting key steps or performing some mistakes, and have the observers record what they see. Compare the results to the “answer key” provided by the trainer. Look for common errors, and remediate with additional training of observers.

Many sectors and disciplines use observation in evaluation. We are interested to hear your experiences and comments regarding challenges and solutions.

Rad Resource – Handouts from Evaluation 2012Our evaluation 2012 roundtable handout expands on this topic.

The American Evaluation Association is celebrating Best of aea365, an occasional series. The contributions for Best of aea365 are reposts of great blog articles from our earlier years. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hello! I’m Tony Fujs, Director of Evaluation at the Latin American Youth Center, a DC based non-profit organization. When asked about the role of my department, I often respond that it aims to be “the GPS unit of the organization”, showing decision-makers whether the organization is on track or not toward achieving its goals.

I like the GPS analogy because it is simple and easy to understand. It also provides an interesting framework to think about performance measurement systems, and how to improve them.

Nonprofits Performance measurement systems

Where we are:                                                                                                                                                                Where we want to be:

Image credit: Biblioteca de la Facultad de Derecho y Ciencias del Trabajo

Image credit: Biblioteca de la Facultad de Derecho y Ciencias del Trabajo

Image credit: Tony Fujs

Image credit: Tony Fujs

 

 

 

 

 

 

 

 

 

 

Let’s look at a few concrete examples to illustrate this point:

Fujs table

Lessons Learned: On data collection.

Recognize that data collection is always a burden: Collect only what is needed. Eliminate manual data entry whenever possible; if not possible make the user interface as intuitive as possible.

On data processing. Automate, automate, automate: Internal evaluators generally work with different instances of the same data sets, therefore data cleaning and other analytical tasks can easily be automated: Use programming tools like Excel macros or R. Modern databases can also be customized to automatically “catch” data entry errors.

Hot Tip: Want to learn more about efficient data processing? I’ll be running a workshop on data management at the next EERS conference.

On providing actionable information

Make sure the information generated by the performance measurement system is useful and understandable for the end user.

Make evaluation results hard to ignore: For instance, they could be displayed on a giant TV screen in the hall of the organization building, so nobody can enter the building without seeing them.

Simple is beautiful

Building a culture of data is often cited as a critical step in generating buy-in toward performance measurement systems. It is a critical step indeed, but partly because performance measurement systems are often perceived as complex and cumbersome by the end-user. Drivers adopted the GPS because it is useful and easy to use, not because they developed a culture of data. Building useful, simple, and intuitive performance measurement systems can also be a powerful and sustainable strategy to generate buy-in.

The American Evaluation Association is celebrating Eastern Evaluation Research Society (EERS) Affiliate Week. The contributions all this week to aea365 come from EERS members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hello, my name is Lindsey Stillman and I work at Cloudburst Consulting Group, a small business that provides technical assistance and support for a number of different Federal Agencies. My background is in Clinical-Community Psychology and so providing technical assistance around evaluation and planning is my ideal job! Currently I am working with several communities across the country on planning and implementing comprehensive homeless service systems. Much of our work with communities focuses on system change by helping various service providers come together to create a coordinated and effective system of care, rather than each individual provider working alone.

Lesson Learned:

  • The new HEARTH legislation includes a focus on system level performance versus program level performance. This has required communities to visualize how each program performance feeds into the overall performance of the system in order to identify how to “move the needle” at a system level. Helping communities navigate between the system level goals and the program specific goals – and the connections between them – is critical.
  • Integrating performance measurement into planning can help communities see the value of measuring their progress. All too often grantees or communities are given performance measures that they need to report on without understanding the links between their goals and activities and the performance measures. Presenting performance measurement as more of a feedback loop can help remove the negative stigma around the use of evaluation results and focus stakeholders on continuous quality improvement.
  • Working with agencies or communities to create a visual representation of the links between processes, program performance and system performance can really help to pull all of the pieces together – and also shine light on serious gaps. Unfortunately many federal grantees have had negative experiences with logic models and so finding creative ways to visually represent all of the key processes and outcomes/outputs/etc. can help to break the negative stereotypes. In several communities we have developed visual system maps that assist the various stakeholders in coming together to focus on the bigger picture and see how all of the pieces fit together. Oftentimes we have them “walk” through the system as if they were a homeless individual or family to test out the model and to identify any potential barriers or challenges. This “map” not only helps the community with planning system change but helps to identify places within the system and processes that measuring performance can help them stay “on track” toward their ultimate goals.

Rad Resources:

The American Evaluation Association is celebrating Atlanta-area Evaluation Association (AaEA) Affiliate Week with our colleagues in the AaEA Affiliate. The contributions all this week to aea365 come from AaEA Affiliate members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello. We are Mindy Hightower King, Research Scientist at Indiana University and Courtney Brown, Director of Organizational Performance and Evaluation at Lumina Foundation. We have been working to strengthen and enhance performance management systems for the last decade and hope to provide some tips to help you create your own.

Lesson Learned: Why is this important?

Funders increasingly emphasize the importance of evaluation, often through performance measurement. To do this, you must develop high quality project objectives and performance measures, which are both critical to good proposals and successful evaluations.

A performance measurement system:

  • Makes it easier for you to measure your progress
  • Allows you to report progress easily and quantitatively
  • Allows everyone to easily understand the progress your program has made
  • Can make your life a lot easier

Two essential components to a performance measurement system are high quality project objectives and performance measures.

Project objectives are statements that reflect specific goals that can be used to gauge progress. Objectives help orient you toward a measure of performance outcomes and typically focus on only one aspect of a goal. Strong Project objectives concisely communicate the aims of the program and establish a foundation for high quality performance measures.

Cool Trick: When developing projective objectives, be sure to consider the following criterion of high quality project objectives: relevance, applicability, focus, and measurability.

Performance measures are indicators used at the program level to track the progress of specific outputs and outcomes a program is designed to achieve. Strong performance measures are aligned with program objectives. Good performance measurement maximizes the potential for meaningful data reporting.

Cool Trick: When developing project measures, be sure to account for the following questions:

  • What will change?
  • How much change you expect?
  • Who will achieve the change?
  • When the change will take place?

Hot Tip: Make sure your performance variables:

  • Have an action verb
  • Are measurable
  • Don’t simply state what activity will be completed 

Rad Resources: There are a host of books and articles on performance measurement systems, but here are two good online resources with examples and tips for writing high quality objectives and measures:  Guide for Writing Performance Measures  and Writing good work objectives: Where to get them and how to write them.

Want to learn more? Register for A Framework for Developing and Implementing a Performance Measurement System of Evaluation at Evaluation 2014.

This week, we’re featuring posts by people who will be presenting Professional Development workshops at Evaluation 2014 in Denver, CO. Click here for a complete listing of Professional Development workshops offered at Evaluation 2014. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Hi everyone!  I’m Yvonne M. Watson, a Program Analyst in U.S. EPA’s Evaluation Support Division (ESD) and Chair of AEA’s EPE TIG.  I’m primarily responsible for managing the division’s internal evaluation capacity building efforts. I read Moneyball: The Art of Winning an Unfair Game by Michael Lewis to understand the importance of a culture that values performance measurement.  The book focuses on the life of Billy Beane, former general manager of the Oakland A’s baseball team and his efforts to use unconventional stats to select members of his team.  Though I did not inherit my grandmother’s passion for baseball, this book helped me gain some important insights that can be applied to the measurement and evaluation of our programs.

Lessons Learned:

  1. Leadership and Using Performance Data and Evaluation Results.  Billy Beane used baseball stats to make decisions about the players that were drafted, traded or let go.  He also demonstrated the critical role of leadership in convincing organizational members to try a new measurement approach.  Our organizations have to take the next step in moving from merely collecting information to routinely analyzing performance data and using evaluation results to inform the decisions we make about our programs. Equally important is the task of asking the right questions and collecting the right data to accurately communicate performance.  These data can either tell a story of fact, fiction, drama, poetry, or comedy.
  2. Data Quality.  Michael Lewis notes that “…inadequate data led the people who ran major league baseball teams to misjudge their players and mismanage their games.”  Similarly, inaccurate and poor quality performance data and evaluation results can lead program managers and decision-makers to misjudge, mismanage, over-value or under-value programs.  Investing in the infrastructure to support good quality performance data and evidence needed for evaluation is crucial.
  3. Insiders and Outsiders. In baseball there are insiders (players, managers) and outsiders (statisticians).  Our organizations and programs can benefit from program insiders who have a wealth of subject matter expertise and knowledge about a program’s goals, clients and underlying assumptions and outsiders – analysts, evaluators etc. external to the program who bring a fresh  perspective at how to measure or evaluate a program.  These individuals can help broaden our thinking about our program’s design, measures, processes and procedures.
  4. Thinking Differently.  Billy Beane exhibited a willingness change things that no longer worked.  In short, he learned to adapt.  Likewise, our organizations – if they are to succeed, must adapt.  We must challenge ourselves not to rely on what we “know or see” but be willing to dig deeper, challenge the status quo and look at our problems, programs and solutions differently.

Rad Resource: Moneyball: The Art of Winning an Unfair Game by Michael Lewis.

The American Evaluation Association is celebrating Environmental Program Evaluation Week with our colleagues in AEA’s Environmental Program Evaluation Topical Interest Group. The contributions all this week to aea365 come from our EPE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

Hi, we’re Eva Bazant, evaluation staff at Jhpiego, and Vandana Tripathi, consultant to global public health programs. Jhpiego is an affiliate of Johns Hopkins University in Baltimore, Maryland, working to improve maternal, newborn and child health globally.

In many sectors, such as education, observations are used for professional evaluation. We are sharing lessons learned from an experience of using structured observation in evaluation of health care quality offered on the day of birth in low-resource settings (our experience was in Madagascar’s hospitals), carried out by the Maternal and Child Health Integrated Program of USAID.

Lessons Learned

  1. Build trust of the individuals being observed and the professionals in charge to allow the observation to happen. Rely on a respected senior colleague to negotiate entry for observers.  Communicate clearly how the data will be used and kept secure, and to whom findings will be disseminated.
  2. Build in enough time to train and standardize observers’ competencies in observation; this can help identify potential challenges with the observation process and tools. Train observers to be a “fly on the wall” and stay long enough to allow employees to feel at ease and act normally, thereby reducing the Hawthorne effect.
  3. Use the shortest checklist/tool needed to cover important topics, to reduce error and fatigue; Validate the tool with topical experts prior to use, and pretest in the field.
  4. Create clear response categories to minimize ambiguity and need for interpretation by observers. Clarify for observers the distinctions between “not observed”, “not done” and “not applicable.”
  5. Interview your observers at the end, and communicate frequently during the process, to document how the observer tools were used. Review cases for completeness and discuss missing data.
  6. Use technology when possible (e.g., smart phone data entry) to increase efficiency in data entry. Ensure observers are comfortable using and maintaining the technology.
  7. Triangulate data from multiple sources to affirm and contextualize observation findings. Observation findings can be compared with interview or inventory data.

Lesson Learned Highlight – Improve validity and inter-rater reliability: During observer training, carry out one or more exercises to promote consistency of data. Have a trainer perform a complex service, omitting key steps or performing some mistakes, and have the observers record what they see. Compare the results to the “answer key” provided by the trainer. Look for common errors, and remediate with additional training of observers.

Many sectors and disciplines use observation in evaluation. We are interested to hear your experiences and comments regarding challenges and solutions.

Rad Resource – Handouts from Evaluation 2012: Our evaluation 2012 roundtable handout expands on this topic.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

I’m Steve Gill, a long-time, independent consultant living in Ann Arbor, Michigan.  I’ve been a mental health therapist, community mental health worker, university professor, program evaluator, small business owner, employee training and development specialist, strategic planner, and management/leadership consultant.

Rad Resource – The Performance Improvement Blog: This is my blog. I post once a week about topics related to increasing learning and effectiveness of leaders and managers in all types of organizations. Often these posts are about evaluation and its application to organizational performance improvement.

Hot Tips – favorite posts: Over the past four years, I have written over 345 posts to my blog. Here are five that I think exemplify my writing about evaluation:

Lessons Learned – why I blog: I blog to learn and to facilitate the learning of others. In order to blog, I must stay current on what’s happening in my areas of interest and what’s happening in current events, as well as reflect on what I’m learning from my own consulting practice. I find it very rewarding to engage others in conversations about these reflections and observations. In addition, my blog is like a personal, professional library.  When I’m investigating a topic, I start by searching my own blog.

Lessons Learned: The blogosphere is frequented by many people who would rather read anonymously and not engage in public conversation (i.e., post comments). My blog has some regulars who often post comments, but there are many more who read my posts and never comment…and that’s okay.

This winter, we’re running a series highlighting evaluators who blog. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

I am David J. Bernstein, and I am a Senior Study Director with Westat. We will be celebrating the 20th Anniversary of the Government Evaluation Topical Interest Group at the 2010 AEA Conference, so I have been reflecting on how government evaluation has changed over the last 20 years. One area that has not changed is how we determine the quality of performance measures for government programs.

Hot Tips: Here are my top 10 list of indicators of performance measurement quality:

10.    Resistant to Perverse Behavior.  Credit goes to the Governmental Accounting Standards Board (1994) for this phrase, which means performance measures should be objective, and not manipulated in a misleading way. Measures that are easily manipulated are less likely to be useful.

9.      Relevant. Performance measures need to be relevant to the government program being measured, or they will not be seen as useful by stakeholders.

8.      Cost-Effective/Affordable. Government managers prefer using resources on programs, not “overhead.” Many managers see performance measures as a “less expensive” evaluation substitute, which it is not since you need evaluation to determine causation. The cost of measurement systems is typically understated, when calculated at all, and systems still need to be affordable.

7.      Accessible, Easy to Capture, Measurable. Measures which are not easy to capture are unlikely to be cost-effective. Evaluation can help identify measures that are linked to program purposes and measurable, hence useful.

6.      Consistent/Reliable. Performance measures should be consistent, because without consistency, comparisons are not possible, and measures will not be usable for tracking program progress.

5.      Comparable. Consistent performance measures allow comparisons with prior performance, benchmarks set by legislatures or executives, or “best practices” by similar organizations.

4.      Results-Oriented. The biggest change in performance measurement in the last 20 years has been an increased focus on results, and performance measures that are results-oriented are seen as being more useful.

3.      Valid, Verifiable, Accurate. We are evaluators, are we not? Performance measures, like evaluation methods, should be valid, verifiable, and accurate, or else they won’t be seen as trustworthy or useful.

2.      Clear/Comprehensible/Understandable. Some government organizations with complex missions and diverse delivery systems such as U.S. Federal government agencies develop multiple complex metrics combining levels of service with percentage of results achieved, making it difficult to judge if programs are really effective. This may make measurement systems technically accurate and politically useful, but the measures themselves may be less useful.

1.      Useful. Performance measurement systems that do not produce useful information will be abandoned. So, with a nod to Michael Quinn Patton, “utilization-focused performance measurement systems” that meet the other quality criteria are more likely to be sustainable and useful in government evaluation and accountability.

The American Evaluation Association is celebrating Government Evaluation Week with our colleagues in the Government Evaluation AEA Topical Interest Group. The contributions all this week to aea365 come from our GOVT TIG members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting Government-focused evaluation resources. You can also learn more from the GOVT TIG via their many sessions at Evaluation 2010 this November in San Antonio.

· · ·

My name is Ted Kniker and I am an executive consultant with the Federal Consulting Group, an organization comprised of federal employees who provide management consulting, executive coaching and customer satisfaction measurement to other federal agencies. Prior to this, I was the Director for Evaluation and Performance Measurement for Public Diplomacy and for the Bureau of Educational and Cultural Affairs at the U.S. Department of State.

One of the questions our customers frequently ask is “how should an organization integrate its evaluation and performance measurement?” This question has become especially relevant to federal agencies as they try to balance improving transparency and performance with mandated reporting requirements.

I explain that I see performance measurement as a compass – it tells us if we are heading in the direction we intended. I think of evaluation as a map – it provides a picture of our terrain, what’s in front of us, behind us, to the sides, and possible paths to reach our destination. Combined, these tools provide a powerful way to provide guidance on our direction, processes, outputs and outcomes.

Lesson Learned – Both evaluation and performance measurement are needed to drive organizational performance: In simplest terms, performance measurement tells an organization what is happening and evaluation provides why it is happening. Reliance on one without the other is similar to driving a car with only two wheels instead of four. I recently came across an equation, presented by Gary Klein, author of Streetlights and Shadows, which I adapted to explain the concept:

Performance = (the reduction of mistakes and variation) + (the increase of insight and expertise)

Performance measurement data in the context of continuous improvement activities, such as Lean and Six Sigma, are used to reduce errors and eliminate waste, and evaluation or assessment feedback, is used to increase our learning to form the basis for sound improvement strategies.

Lesson Learned – When not integrated, Evaluation and Performance Measurement tend to become compliance instead of learning activities: In my experience, managers who don’t evaluate are left without the tools to explain challenged performance, often leading to ineffective blame and shame performance management systems. Managers who evaluate without monitoring performance generally have evaluations that end up as credenza-ware.

Hot Tip: Align evaluation and performance measurement by using each to reinforce the other, as appropriate, in management systems and evaluation projects. At State, we integrated performance measurement and evaluation by including the key research and survey questions used to gather performance data in most, if not all, of our evaluations. This not only helped to verify the performance results, it allowed us to deeply explore how and why we were achieving specific results, so they could be reported and replicated.

The American Evaluation Association is celebrating Government Evaluation Week with our colleagues in the Government Evaluation AEA Topical Interest Group. The contributions all this week to aea365 come from our GOVT TIG members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting Government-focused evaluation resources. You can also learn more from the GOVT TIG via their many sessions at Evaluation 2010 this November in San Antonio.

·

Archives

To top