I am David J. Bernstein, and I am a Senior Study Director with Westat. We will be celebrating the 20th Anniversary of the Government Evaluation Topical Interest Group at the 2010 AEA Conference, so I have been reflecting on how government evaluation has changed over the last 20 years. One area that has not changed is how we determine the quality of performance measures for government programs.
Hot Tips: Here are my top 10 list of indicators of performance measurement quality:
10. Resistant to Perverse Behavior. Credit goes to the Governmental Accounting Standards Board (1994) for this phrase, which means performance measures should be objective, and not manipulated in a misleading way. Measures that are easily manipulated are less likely to be useful.
9. Relevant. Performance measures need to be relevant to the government program being measured, or they will not be seen as useful by stakeholders.
8. Cost-Effective/Affordable. Government managers prefer using resources on programs, not “overhead.” Many managers see performance measures as a “less expensive” evaluation substitute, which it is not since you need evaluation to determine causation. The cost of measurement systems is typically understated, when calculated at all, and systems still need to be affordable.
7. Accessible, Easy to Capture, Measurable. Measures which are not easy to capture are unlikely to be cost-effective. Evaluation can help identify measures that are linked to program purposes and measurable, hence useful.
6. Consistent/Reliable. Performance measures should be consistent, because without consistency, comparisons are not possible, and measures will not be usable for tracking program progress.
5. Comparable. Consistent performance measures allow comparisons with prior performance, benchmarks set by legislatures or executives, or “best practices” by similar organizations.
4. Results-Oriented. The biggest change in performance measurement in the last 20 years has been an increased focus on results, and performance measures that are results-oriented are seen as being more useful.
3. Valid, Verifiable, Accurate. We are evaluators, are we not? Performance measures, like evaluation methods, should be valid, verifiable, and accurate, or else they won’t be seen as trustworthy or useful.
2. Clear/Comprehensible/Understandable. Some government organizations with complex missions and diverse delivery systems such as U.S. Federal government agencies develop multiple complex metrics combining levels of service with percentage of results achieved, making it difficult to judge if programs are really effective. This may make measurement systems technically accurate and politically useful, but the measures themselves may be less useful.
1. Useful. Performance measurement systems that do not produce useful information will be abandoned. So, with a nod to Michael Quinn Patton, “utilization-focused performance measurement systems” that meet the other quality criteria are more likely to be sustainable and useful in government evaluation and accountability.
The American Evaluation Association is celebrating Government Evaluation Week with our colleagues in the Government Evaluation AEA Topical Interest Group. The contributions all this week to aea365 come from our GOVT TIG members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting Government-focused evaluation resources. You can also learn more from the GOVT TIG via their many sessions at Evaluation 2010 this November in San Antonio.