Hello, we are Rashon Lane, Alberta Mirambeau from the Centers for Disease Control and Prevention, and Steve Sullivan from Cloudburst Consulting and we work together on an evaluation aimed at assessing the uptake, use and impact of national public health hypertension recommendations by the Institute of Medicine (IOM). If you’ve ever wondered how to assess if public health programs are shifting their priorities to address evidence-based recommendations you might consider a methodology we used entitled alignment scoring analysis. In short, an alignment scoring analysis is a type of content analysis wherein narrative descriptions of organizational activities are analyzed to determine whether they support specific goals or strategies. We conducted a pre-post alignment scoring analysis of state health department work plans to objectively determine if their project portfolios align with nationally recommended priorities.
Lessons Learned:
- Conduct pre-post content analysis. During our content analysis we coded state work plan activities as aligned, mis-aligned or neutral to the IOM recommendations. As a result, we were able to share with program stakeholders that many state health departments were able to adjust their prevention priorities within 18 months to reflect national priorities. If you are working on an evaluation to assess changes in priorities over time, you might consider conducting a similar pre-post content analysis to determine the degree to which public health programs align with priorities and how these priorities change over time.
- Use stringent criteria. Use stringent criteria to consider activities as aligned, mis-aligned or neutral for more accurate coding.
Hot Tips:
- Use a database. Use a database to facilitate the review of documents being analyzed and to speed reporting. If you plan to use multiple reviewers, be sure to keep track of which reviewer coded a document so you can check inter-rater reliability and improve training on your coding protocol.
- Use alignment scoring. Use alignment scoring analysis results to provide recommendations to program stakeholders on how they might shift priorities that are NOT aligned with national recommendations that have proven to be effective.
Resources:
- Learn more about our evaluation in the article Epub ahead of print: Applying the Interactive Systems Framework to the Dissemination and Adoption of National and State Recommendations for Hypertension
- Learn more about alignment theory in Venkatraman’s seminal 1989 article: The Concept of Fit in Strategy Research: Toward Verbal and Statistical Correspondence
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.