Hi, I’m Lisa Melchior, President of The Measurement Group LLC, a consulting firm focused on evaluation of health, behavioral health, and social services for at-risk and vulnerable populations. I’m delighted to share some thoughts about the role of federal performance data and reporting requirements as they pertain to local evaluation. Since the inception of the Government Performance and Results Act (GPRA) in 1993, my colleagues and I have conducted dozens of evaluations that involved complying with these data collection and reporting requirements.
GPRA is U.S. federal law which created a mechanism for government agencies to engage in strategic planning, performance planning, and performance reporting. In 2010, the GPRA Modernization Act was updated, emphasizing priority- setting and cross-organizational collaboration. It also increased its focus on using goals and measures to improve outcomes of federally funded programs.
Many of the programs my colleagues and I evaluate are funded by the US Department of Health and Human Services, Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Substance Abuse Treatment (CSAT), which is required to report to Congress on progress on its designated GPRA measures. SAMHSA/CSAT requires organizations that provide direct client services funded by CSAT discretionary grants to implement the Client Outcome Tool. Although it can vary depending on the grant program, grantees are generally required to collect this information from individuals at program intake, at 6-months post-intake, and at program completion or discharge. The GPRA tool includes questions regarding the participant’s housing situation, physical and mental health, substance use, sexual activity and criminal justice system involvement. While this standardizes data collection for all grantees within a funded initiative (and to a broader extent across all the funding agency’s discretionary grantees), it also has a substantial impact on evaluation design, implementation, and analysis at the local level.
Hot Tip:
GPRA data can be used for program monitoring and evaluation at the local level. Grantees and their evaluators can access summary reports of selected GPRA data (SAMHSA grantees use a system called SPARS). Grantees also have the capability to download their raw data for local use.
Cool Tricks:
Try to integrate required performance measurement data collection protocols with additional measures used for project-specific evaluation. Aim to minimize repetition and redundancy. Pay close attention to the flow of when and how questions are asked of program participants.
Lessons Learned:
While GPRA data is useful for local evaluation, it is often not sufficient for this purpose. For example, self-reported substance use is only measured in the past 30 days. Mental health symptoms are also measured by responses to single-item self-report measures of past 30-day symptoms. If it is important to document lifetime substance use patterns, or to know about actual DSM-based mental health conditions, it is necessary to supplement the GPRA measures.
Rad Resources:
General information: https://en.wikipedia.org/wiki/Government_Performance_and_Results_Act
https://obamawhitehouse.archives.gov/omb/mgmt-gpra/index-gpra
Information about SAMHSA’s GPRA Measurement Tools: https://www.samhsa.gov/grants/gpra-measurement-tools
The American Evaluation Association is celebrating Behavioral Health (BH) TIG Week with our colleagues in Behavioral Health Topical Interest Group. The contributions all this week to aea365 come from our BH TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hi Lisa,
I enjoyed reading your post; in my work, I support health care programming, and I’m also currently working toward a Master’s degree in education, with a focus on evaluation. With these two professional interests, your post really spoke to me.
I live in Canada, and so am not familiar with the GPRA, but you offer some great advice that evaluators can benefit from, regardless of where they are located. If you have any additional advice for my particular evaluation context (described below), I would love to hear it!
In my current course, we’re creating an evaluation design, and we’re just getting to the stage where we are describing our data collection methods. As such, your ‘Cool Tricks’ are especially relevant!
I’ve included the Most Significant Change (MSC) approach in my evaluation design. Since the MSC is all about gathering stories (which may not be entirely representative of the entire group’s experience with the program), I have also included more traditional quantitative and qualitative evaluation methods in my design (McDonald, Stevens & Nabben, n.d.).
Your comment made me realize that I need to focus on the ‘significant’ and exceptional in selecting the stories generated by stakeholders. The traditional qualitative/quantitative data collection methods will provide the representative data of the program. The stories coming from the MSC portion of the evaluation, however, should focus on the very best (or very worst) aspects so that I can get a sense of what the program truly can be, or how it is missing the mark.
I now realize that the MSC aspect of this evaluation is all about ‘setting the bar’ and should not attempt to gather the same type of data that the more traditional evaluation methodologies will collect. Doing so would make me guilty of being repetitive and redundant, and a real missed opportunity to learn about the truly exceptional aspects of the program.
I also appreciate your section on ‘Lessons Learned.’ It’s a reminder that evaluators may need to seek out additional information when the data provided isn’t enough. As I progress throughout this program, I’m learning that successful evaluations are those that tell a complete and accurate story, which can result in meaningful change; thank you for the reminder that evaluation is more than just reporting on available results!
Reference
McDonald, B., Stevens, K. & Nabben, T. (n.d.) Most Significant Change. Better Evaluation. Retrieved from https://www.betterevaluation.org/plan/approach/most_significant_change