Kristi Fuller and Glenn Landers on Ensuring Utility in Evaluation Practice

Hello all! We are Kristi Fuller and Glenn Landers, staff at the Georgia Health Policy Center. The Center is housed within Georgia State University’s nationally ranked Andrew Young School of Policy Studies and provides evidence-based research, program development, and policy guidance.

We gave a roundtable presentation at the American Evaluation Association’s (AEA) 2012 conference focused on ensuring utility in evaluation practice, in which we used the Centers for Medicare & Medicaid’s Money Follows the Person (MFP) demonstration program as an example.  Our current evaluation of the MFP program for the state of Georgia has the potential to last ten years.

Hot Tip: When conducting an evaluation over a long time frame, it is conceivable to get into a pattern and produce reports in which stakeholders begin to lose interest. However, keeping the Joint Committee on Standards for Educational Evaluation (JCSEE) Program Evaluation Standards regarding utility in focus can help evaluators avoid this trap.

Lessons learned:

1. Utility standard 2 emphasizes the importance of devoting adequate attention to all relevant stakeholders. For MFP, regular evaluation steering committee meetings bring diverse perspectives of those interested in results, as well as those impacted by the program. Through this interaction, we gain important information used to plan the evaluation so that it provides benefits to a broad range of stakeholders including program participants, familial advocates, attorneys providing legal assistance, programmatic staff, and nursing facility advocates.
2. Utility standard 5 discusses the importance of providing information relevant to needs that are both known and evolving. Recognizing that as programs develop and grow the needs of the invested parties also change is important for ensuring that what is being studied continues to be of relevance to stakeholders. In our experience with MFP, we’ve found that program personnel are interested in delving into data to understand their clients’ experiences, whereas the state’s Medicaid agency is particularly concerned about how services are being utilized.
3. Utility standard 6 describes utilizing various communication methods to create processes and products that are meaningful for challenging and reinterpreting understandings. Interpretation of data can be done in a myriad of ways, and AEA’s Data Visualization and Reporting TIG provides great ideas. One way that we’ve tried to manage this is through dropping the production of our full report from quarterly to semi-annually, allowing more time to develop dashboards and ad-hoc analyses.

Rad Resource:

Food for Thought:

• What are you doing that works well regarding how you engage stakeholders?
• How are you managing different points of view successfully?
• What do you think works well with your data presentation?
• What could you do either more of or differently?

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.