EPTF Week: Affecting Policy in the Public Sector by Tom Chapel

I’m Tom Chapel. I’m a current AEA Board member and also serve in a senior evaluation position with a Federal agency.  I’ve been consulting on evaluation and how to incent evaluation capacity in all types of organizations for several decades. I’m not sure that setting evaluation policy in general is significantly different than setting evaluation policy in any large organization, and I hope these hints below will resonate across sectors.  Here are some tips to devising, implementing, and putting an effective evaluation policy to use, based on my primarily public sector experience.

Hot Tips:

Be flexible about designs.  Unless all evaluations in the organization are done to demonstrate impact using a control group, then be clear in the policy that evaluation designs can vary with the situation.  A process evaluation does not require the rigor of an impact evaluation.  High-level evaluations for management feedback may benefit as much from a dashboard of performance measures as from a full-on research design.  Regardless, all of it is evaluation.

Commit yourself to continuous program improvement. A useful policy makes clear that evaluation plays a role at all stages of program development, even if the key evaluation questions and methods vary over the life of the program.

Expect pushback. There are many reasons programs resist evaluation—policy or not—besides the fear of exposing program failure. Diverting resources from the program, the long timeframe for results, and, sometimes, the lack of external validity are but a few.  But if you’re faithful to the first few steps in designing and implementing your policy—designs that match the situation and commitment to an evaluation that yields useful results—then most of the reasons to resist or “game” the policy disappear.

Be  “high touch”—the usefulness of technical assistance.  No one likes unfunded or neglected mandates.  An evaluation policy that comes without resources or technical assistance is unlikely to take hold.  Evaluation is not nearly as hard as people make it, and the purpose of technical assistance is both about keeping people from overkilling (too much attention) or underkilling (not enough attention).  Coaching them helps them get on track from the start.

Look for “process use” wins.   Policy gets evaluation in the door.  But an evaluation process that provides clarity or uncovers inconsistent or logical gaps in the program design or theory of change is often the “aha” needed to sell evaluation to the skeptical. It should be no surprise that the biggest added value of evaluation may come before the data are collected.

Standardize your terms.  Nothing undermines evaluation, performance measurement, and even strategic planning than inconsistency in definition of terms.  Whether you require that people adopt your definition or not, establish definitions for how key terms—input, output, outcome, impact, indicator, and measure—will be used in the policy.

Setting organizational policy of any kind is hard.  When that policy requires more resources or a shift in traditional practices, it becomes all the harder.  But paying attention to these six tips might pave the road to success.

The American Evaluation Association is celebrating AEA’s Evaluation Policy Task Force (EPTF) week. The contributions all this week to aea365 come from members of AEA’s EPTF. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.