I am David Bernstein, owner of DJB Evaluation Consulting and Director of Evaluation with the General Commission on Religion and Race of the United Methodist Church. Don’t let these non-governmental positions fool you. I have been involved in government evaluation for many years, and was the founding Chair of AEA’s Government Evaluation TIG.
I have been reflecting on government evaluation, and have five “lessons learned” I would like to share.
First, benchmarking gets government performance measurement closer to evaluation since it includes a comparison. However, performance measurement is NOT evaluation, but instead complements and contributes to it. You can measure without evaluating, but you cannot evaluate without measuring.
Second, I’ve long advocated that using government performance measures is much like evaluation utilization. Patton’s conception of primary evaluation uses (Utilization-Focused Evaluation) is similar to Epstein’s conception of primary uses of government performance measures (Using Performance Measurement in Local Government).
Third, the U.S. Paperwork Reduction Act (PRA) requires that even small data collections of 10 or more be reviewed. Reviews should be based on hours of burden, not the number of respondents. Rad Resource: I provided comments to the AEA Evaluation Policy Task Force, which developed a position paper on improving the PRA. It’s a great example of how evaluators can contribute to evaluation policy discussions. Check out the statement in the AEA archives.
Fourth, logic models can be a useful tool in government evaluation. Hot Tip: It is OK to create a “logic chart,” linking all activities to all outputs to all outcomes, but then take it to the next step. Each activity needs clear linkages to results (outputs, outcomes, outcomes, and impacts). The Kellogg Foundation has some useful tools on logic models.
Fifth, are random control trials or RCTs the gold standard for measuring effectiveness? Maybe, but not the only one. I summarized 28 criteria from the evaluation literature to see if an RCT is feasible and presented these in an Ignite presentation at the 2013 AEA Conference.
My career just started yesterday, and this is my 36th year. As the late Warren Zevon said, “Enjoy every sandwich” (The Late Show with David Letterman, October 30th, 2002).
The American Evaluation Association is celebrating Gov’t Eval TIG Week with our colleagues in the Government Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Gov’t Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.