Hello! We are Tarek Azzam (Claremont Graduate University) and Matt Keene (Environmental Protection Agency). We are members of the External Review Panel for ECLIPS.

Whoa! Even though it’s oh-so-tempting to try, you don’t need the perfect string of words to define “systems-oriented evaluation.”

Lesson Learned: It’s in the roots of evaluation 

At its core a systems approach to evaluation encourages the evaluator to consider the physical, political, and structural issues that surround a program, and to examine how these factors help or hinder the success of a program. This examination and reflection process is incorporated in the work and writings of lead evaluation scholars such as Lee Cronbach, Robert Stake, and Jennifer Greene. The presence of systems thinking also can be seen in our standards (specifically standards F3, A4, and A7).

And it’s also something different, because it requires the evaluator to recognize that the program is part of systems that have their own dynamics. It forces the evaluation to examine issues that go beyond the process and outcomes of a single program.

Lesson Learned: How to become a systems-oriented evaluator

1)      Adopt some habits of systems thinkers…

2)     Know the domains of Social Ecology and use them to understand the leverage points of boundaries, relationships, and perspectives. Donella Meadows says that boundaries are problem dependent and messy. Don’t make the world linear for your mathematical or administrative convenience.

3)     Delve into the dynamics of systems to find the regions of organized, adaptive, and unorganized patterns.

4)     Find leverage points (places to intervene where small tweaks can lead to big changes). Here are some leverage points from Meadows you can use to influence relevant systems:]

5)     Let systems thinking do fuzzy things to your logic model. A fuzzy logic model takes into account the dynamic nature of the systems surrounding a program. It gives a visual image of the complexities that can affect processes and outcomes.

In the ECLIPS, all of our logic models used to look like this:

But after applying systems thinking, we made them into fuzzy logic models. See how different they look.

Try creating a fuzzy logic model to find and depict the system’s complexity, making your logic model more useful to people for a longer time.

That’s all for now

This concludes ECLIPS week on aea365! Don’t expect to learn about systems alone or in a short period of time. It may well be a journey into a new way of thinking about evaluation. Get involved in an existing community of practice about systems or form your own group. ECLIPS members are happy to share what they are doing.

The American Evaluation Association is celebrating this week with our colleagues involved in ECLIPS—Evaluation Communities of Learning, Inquiry, and Practice about Systems—and the AEA Systems in Evaluation Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! We are Karen Peterman (Karen Peterman Consulting, Co.) and Marah Moore (Director of i2i institute and ECLIPS coPI). We are going to talk about some of the implications of a systems approach for the role of the evaluator.

In a recent Thought Leaders Forum, Michael Patton reminded us that evaluation is a transdisciplinary field. Evaluators need expertise in evaluation theory, practice, methods and use, as well knowledge of theories of change and how to generalize what they have learned about patterns in effective interventions. A systems approach to evaluation can enhance evaluators’ work across each of these areas of expertise.

I (Karen) found the ECLIPS’ biggest impact on me was in how I view my overall role as an evaluator. And I (Marah) found that the evaluator’s role was revisited throughout the process of addressing each new systems concept. Here are some ideas we came away with:

Lesson Learned: Use your “systems-based evaluation expertise” to add value for clients.

Systems concepts can provide a valuable lens through which to view project evaluation findings. For example, you can help clients consider their individual projects within the context of academic literature, their institution’s larger mission, and/or their funders’ goals.

To guide these discussions, check out ZIPPER, A System-Based Evaluation Mnemonic and the Systems Archetypes.

Lesson Learned: Use a systems approach to push evaluation beyond the traditional stages.

The ECLIPS graphic below shows a traditional and a systems view of the stages of evaluation.

The systems view encourages the evaluator to work at the intersection of the traditional stages. The evaluator asks clients and participants to help shape:

• the evolution of the evaluation plan,
• the data collection procedures, and
• the interpretation of results.

Bringing together a systems orientation and participatory evaluation leads to an evolution in evaluation practice.

Hot Tip: Break down the silo approach to evaluation.

At AEA last fall, a number of presenters stated that the purpose of their work was to improve education. The systems perspective provides methods to start thinking about and achieving that goal on a broader scale. It enhances the thinking and work of evaluators by providing a framework and tools to move beyond the evaluation of the immediate project to start challenging clients (gently!) to think about system-level change and how their projects really can make a big-picture difference.

Read the final ECLIPS blog tomorrow where Tarek and Matt give an overview of systems- oriented evaluation and look at some fuzzy logic models.

Rad Resources:

• Karen’s presentation at the 2012 AEA conference:Blurring the Lines: The Process of Applying Systems Thinking to an Ongoing Evaluation.
• ECLIPS on the InSites website

The American Evaluation Association is celebrating this week with our colleagues involved in ECLIPS—Evaluation Communities of Learning, Inquiry, and Practice about Systems—and the AEA Systems in Evaluation Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello, we are David Reider (Education Design, Inc.), Ginger Fitzhugh (Evaluation and Research Associates), and Alyssa Na’im (Education Development Center, Inc.). As ECLIPS members, we are incorporating systems concepts into STEM education evaluations related to the National Science Foundation program, Innovative Technology Experiences for Students and Teachers (ITEST).

Keying off the iceberg diagram (see Monday’s post), we go deeper into a system to find leverage points for change by considering:

• boundaries (demarcations that define regions or entities)
• relationships (connections and exchanges between project parts or people)
• perspectives (the paradigms held by various parties and the purposes they seek)

Hot Tip: Don’t disregard the simple in a complex setting.

I (David) am evaluating a project using science probes and models in K-12 classrooms in four states. Although there were vastly different support structures in the sites, one of the lessons I learned in my evaluation was quite simple. The more frequently teachers posted to the online learning platform (thus reaching beyond the boundaries of their classroom), the higher the quality of their classroom projects. This was a case of leveraging small actions toward larger gains.

Hot Tip: Ask about boundaries, relationships and perspectives.

Questions That Matter has terrific examples of evaluation and interview questions that relate to boundaries, relationships and perspectives. I (Ginger) added several of these questions to our interview protocols for project leaders. For example, we added, “What, if any, unanticipated outcomes (positive or negative) have happened in the project thus far?” We learned that parents were interested in obtaining the project equipment to use with their children. This spurred the project team to consider how to make the materials more widely available.

Hot Tip: Consider how program goals can be leveraged.

I (Alyssa) am now paying more attention to acknowledging and identifying the boundaries, perspectives, and relationships for both program implementation and evaluation. Programs express their perspective through their statement of purpose, e.g., improving the nation’s STEM workforce development capacity. Systems thinking helps us understand why their strategies to accomplish this purpose may overlap or diverge and to see possible leverage points for change.

Hot Tip: Don’t lead with systems language; find ways to include it.

It’s not always necessary to say, “This is a systems idea.” Rather, use familiar language to explain how what you are doing adds value to the evaluation and the project.

Join us again tomorrow as we move away from the Iceberg diagram to consider the role of evaluators.

Rad Resources

• Ginger’s 2012 AEA presentation, Challenges and Successes Associated with Introducing Systems Concepts to an Existing Evaluation, is about applying boundaries, relationships, and perspectives to evaluations.
• David’s 2012 AEA presentation, Leverage Points: Places to Intervene in a System.
• ITEST LRC website

The American Evaluation Association is celebrating this week with our colleagues involved in ECLIPS—Evaluation Communities of Learning, Inquiry, and Practice about Systems—and the AEA Systems in Evaluation Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! We are Lloyd Bingman (Brighter Day Consulting, LLC and evaluator of an NSF STEM project) and Pat Jessup (InSites associate and coPI for ECLIPS). We’re here to talk about the “Patterns” portion of theiceberg diagram displayed on Monday of ECLIPS week.

Both of us evaluate programs that are part of complex social systems that have multiple and dynamic patterns. An application of complexity theory provides us with a way to understand and distinguish among three patterns of system dynamics:

• Organized patterns: With high agreement and high certainty in the system, the patterns appear fairly predictable.
• Self-organizing: With a middle range of agreement and certainty, the patterns are adaptive.
• Unorganized patterns: With low agreement among key players and low certainty, patterns may not be present.

Creating a Robot Diagram to Understand Coordinator’s Role in a Complex System

When we talked about systems dynamics in the ECLIPS, I (Lloyd) discovered a systems lens for evaluating a project I was working on. I envisioned a key project leader superimposed on the diagram of systems dynamics. The leader is attempting to connect all of the project parts with their different dominant patterns of movement.

In this illustration, the “robot” figure is the IT Apprenticeship Coordinator. The Coordinator brings all the pieces together to meet program goals. The numbers on the robot correspond to different program goals.

At the bottom left (#1), the goal of presenting the apprenticeship program is controlled and organized; the Coordinator has direct control of presentations. The other goals on the left side (#3, 5) tend to be fairly predictable but key players do not agree as much on these goals as on the #1 goal.

On the right side of the robot, the activities related to the three goals are more self-organizing or unorganized. The  college recruitment process goal (#2), bottom right side, is less certain than #1 because the college competes with other colleges at recruitment events. Moving up the right side, the approaches to reaching the goals (#4, 6) are increasingly unorganized.

At the head of the diagram (#7), the Coordinator is constantly planning, assessing, and implementing activities to ensure project success.

Insights Gained from the Robot Diagram

This image provided new insight into the relationships among the parts and where changes could lead to a more networked flow of information. For example, by strengthening the relationship with the state liaison (#4), I (Lloyd) was able to gain data on the state workers’ experiences with the apprenticeship program.

Join us tomorrow when Dave, Ginger, and Alyssa talk about identifying leverage points for changing a system.

Rad Resources

• Using Complexity Science Concepts in Evaluation
• Designing initiative evaluation: A systems-oriented framework  

The American Evaluation Association is celebrating this week with our colleagues involved in ECLIPS—Evaluation Communities of Learning, Inquiry, and Practice about Systems—and the AEA Systems in Evaluation Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! I’m David Hata, an independent consultant who serves as an external evaluator for a number of NSF-funded Advanced Technological Education (ATE) projects and centers.

I’m here to share my experience helping a STEM project see itself as part of a system that creates value. For some clients, success is doing a lot of activities. The clients may not be thinking about what value those activities create.

My early evaluations focused on evaluating individual activities using methodologies such as described in Kirkpatrick’ book Evaluating Training Programs: Four Levels. With my recent participation in ECLIPS and the Synergy Project, I have started to think of evaluation more holistically in terms of systems and value creation. I now view NSF ATE projects and centers as systems that create value at multiple places within their structure. By recognizing these value-creation systems, I am discovering more ways that evaluation can help fulfill the mission of the ATE program at NSF—to increase the number of and strengthen skills of technicians for the workforce through the implementation of workforce development initiatives.

Lesson Learned: Finding Value in the Interconnections  

A useful conceptual framework for assessing value creation has been developed by Etienne Wenger and colleagues.

They outline five types of value: immediate, potential, applied, realized, and reframing. I used the first four types of value to help develop a road map for The Southwest Center for Microsystems Education (SCME), an ATE regional center at the University of New Mexico in Albuquerque. Working with my client to create the road map helped them develop a systems view of their project. The road map shows the connections between grant activities and a career pathway that produces advanced technicians for the U.S. workforce.

The diagram below shows how SCME activities fit into a career pathway from high school to community college to workforce.

Center activities can be viewed as value investments:

• immediate value: knowledge, skills, and materials gained by each participant;
• potential value: what each teacher plans to do with their new knowledge, skills, and materials;
• applied value: what changed in classroom instruction and student learning;
• realized value: number of microsystem technicians produced based on graduate data.

The evaluation measures the value created by these investments at different points in the system. The links in the diagram emphasize the nature of the exchanges. Understanding the links between the boxes is as important as defining the activities and outcomes in the boxes.

Check out the following resource and join us tomorrow as my fellow ECLIPS members examine system patterns via a Robot diagram.

Rad Resource: Donella Meadows’ book, Thinking in Systems.

The American Evaluation Association is celebrating this week with our colleagues involved in ECLIPS—Evaluation Communities of Learning, Inquiry, and Practice about Systems—and the AEA Systems in Evaluation Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Greetings! We are Beverly Parsons (InSites and ECLIPS principal investigator) and Veronica Thomas (Howard University and ECLIPS external advisor). Today we introduce you to an exploratory research project, Evaluation Communities of Learning, Inquiry, and Practice about Systems (ECLIPS). It’s funded through a National Science Foundation grant to InSites.

The ECLIPS Community of Practice is 15 evaluators involved in STEM (science, technology, engineering, and math) education evaluations. During 18 months of webinars and annual meetings, ECLIPS members discussed and applied system concepts—especially systems dynamics and complex adaptive systems concepts—to their work. This week we share examples of our learning through this exploratory project.

Lessons Learned: A Systems-Oriented Evaluation?

We’re using a definition of a system from Meadows, Thinking in Systems: “A system is an interconnected set of elements that is coherently organized in a way that achieves something.”What the system achieves may or may not be what we want.

ECLIPS members identified the systems of relevance to their work. They looked at patterns within the systems, paying particular attention to culture and social justice – two areas that I (Veronica) kept in the forefront of the ECLIPS work.

In this type of systems-oriented evaluation, we pay attention not only to activities and results but also to patterns; norms, infrastructure, and policies; and paradigms. You’ll hear more this week about how ECLIPS members have used the iceberg diagram to go deeper in their thinking about systems.

Hot Tip:

Use a systems lens in your evaluation practice to:

• ask different kinds of evaluative questions, including questions that address social justice concerns (e.g., questions about access and opportunity)
• look for patterns that give clues about appropriate theories of change
• find leverage points (where small changes can create large effects),
• consider different roles as an evaluator (e.g., to include social change agent)

This Week with ECLIPS

Tuesday: Dave describes a systems view of a project in the form of a “road map” that shows where interconnections create value.

Wednesday: Lloyd and Pat provide an example of seeing and understanding patterns in systems.

Thursday: David, Ginger, and Alyssa discuss working with boundaries, relationships, and perspectives as leverage points for change.

Friday: Karen and Marah address how a systems orientation influences an evaluator’s role.

Saturday: Tarek and Matt give a wrap-up of systems concepts and connections to fuzzy logic models.

At another time, we’ll share our learning about how to use the systems concepts in more powerful ways than we could do in this exploratory project.

Rad Resources

• ZIPPER, A System-Based Evaluation Mnemonic
• Using Complexity Science Concepts in Evaluation
• Designing initiative evaluation: A systems-oriented framework
• Thomas, V.G. (2011). “Cultural issues in evaluation,” American Journal of Evaluation, 34 (4), 578-582.

The American Evaluation Association is celebrating this week with our colleagues involved in ECLIPS—Evaluation Communities of Learning, Inquiry, and Practice about Systems—and the AEA Systems in Evaluation Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! I’m Eric Sarriot and I am the Director of the Center for Design and Research in Sustainability (CEDARS) at ICF International. I would like to introduce you to the CEDARS blog, on Sustainable Human and Social Development, its planning and evaluation (definitely within a Global/International Development perspective).

Rad Resource – Sustainable Human and Social Development (http://cedarscenter.blogspot.com/): It’s an ongoing discussion about anything and everything related to taking sustainable health development seriously, and what this changes in the way we ‘do business’ or try to learn. After decades of talking, some say to myself, but mostly to friends and colleagues in the health development community, I’d like to expand the conversation to include anyone interested in complex systems’ evaluations, or simply way to learn to do things differently and better in global development. New content is posted about once or twice a month. More info about CEDARS is here.

Hot Tips – favorite posts: The blog has been operating for over a year now, and has attracted commentary on topics like climate change and adaptation, food security, transition to country ownership and others. A couple of my favorites are:

• Projects Don’t “Do” Sustainability, Do They? – Trying to advance sustainability, we run into a common criticism: “I like these ideas; it makes sense, and it does or would make sense to our local partners. But really, that’s not how we work. We have 30 to 60 days to write a proposal, then staff up and kickoff a project and get deliverables. When exactly would you introduce those ideas?” This post addresses this question….
• Emergence of Sustainability in a Complex System – Not just because you can see me on a Youtube video! This post and video link is about a conversation between health and food security evaluation professionals. Panelists discuss how sustainability can be defined and addressed practically in complex adaptive systems.

Lessons Learned – why I blog: I blog because evaluation of sustainability takes the conversation further than single programmatic sectoral and blogging invites more people in. The purpose of the blog is to inform and provoke discussion. It is an outlet for people concerned about the big social development issues of our time and who want to help all of us challenge our practice, whether working in the ‘North’ or the ‘South’.

Lessons Learned – what I’ve learned: You really have to reach beyond your tried and trusted professional network and community in order to learn new things (remember the ‘strength of weak ties?’). That’s also why I’d like to get involved in the AEA Systems in Evaluation TIG. Come and visit CEDARS!

This winter, we’re continuing our series highlighting evaluators who blog. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

My name is Pat Seppanen.  I’ve worked as an evaluator now for more than 20 years.  A good part of my practice has centered on evaluating complex change—initiatives designed to address human needs and community problems that do not fit into established program and policy categories.  If concepts like non-linearity, emergence, adaption, uncertainty, and coevolution describe the work of your clients, I bet you have been having a tough time applying traditional evaluation designs.  I’d like to offer a few tips and resources regarding evaluation of complex systems, hoping that you can use them in your work.

Hot Tips:

1. Evaluation and consultation are inseparable in complex change initiatives and need to be integrated.   In my experience the work is 50% evaluation and 50% consultation to facilitate successive iterations of collaborative problem solving and mutual learning.  Since I am strongest on the evaluation side, I have joined up with someone who is an experienced facilitator and coach to do work. 
2. A situational analysis is a vital step to getting the work going.   I usually build a situational analysis in as an activity in my work plan.   The information you assemble will help you see patterns that will inform your design.  For example, for one citywide initiative we are working on, the evaluation design is organized by the major buckets of work and different types of data are generated based on the information needs of stakeholder groups operating at different levels of the system.  In another initiative, we have organized data collection in terms of the different levels of the system:  national, community, center, program, and individuals.    In doing a situational analysis, I use versions of Human Systems Dynamic tools developed by Glenda Eoyang (see the resource below).
3.  A complex initiative may include components that are simple, complicated, and complex (see Chapter 4 of Patton’s book that I’ve listed as a resource below for a great discussion of the characteristics of these components)—some components may not benefit from developmental evaluation while others will.  If the focus is on improving and stabilizing the component, then a formative evaluation design is needed.  But if the focus is on “learning by doing” then I’d propose a developmental evaluation design

Rad Resources:

I look to Michael Quinn Patton’s book, Developmental evaluation:  Applying complexity concepts to enhance innovation, as my primary resource.

A much shorter monograph by Hallie Preskill and Tanya Beer titled Evaluating social Innovation very concisely offers lessons about doing evaluation to support adaptation.

If you are looking to learn more about complex adaptive change I recommend looking into Human Systems Dynamics.

The American Evaluation Association is celebrating Minnesota Evaluation Association (MN EA) Affiliate Week with our colleagues in the MNEA AEA Affiliate. The contributions all this week to aea365 come from our MNEA members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

complex adaptive systems · situational analysis

Hello, I am Brian Pittman, a Research Associate at Wilder Research. Our work involves many different topics, scopes, and stakeholders, but an increasing proportion of our projects deal with complexity. Therefore, we work to learn and use the principles and practices of complex systems in our work. This post includes a brief primer of the concepts, but it is not intended to be a thorough explanation of complex systems theory.

Hot Tip: Identifying complexity.

First, it is important to understand when you are dealing with a complex system. The three primary characteristics of complex systems are:

• Openness. Complex systems include many inter-related and interacting entities (including other systems) that are scalable (agent affects the system and system affects agent) and co-evolving.
• Diversity. Complex systems have diverse and varying types of entities or agents.
• Uncertainty. Unpredictable may, and often do, occur.

These characteristics help to define systems capable of emergence. Now you may be recognizing that some of your projects are dealing with complex systems, or even the projects themselves are complex systems! Next, let’s look at some of the considerations for engaging complex systems.

Hot Tip: Key considerations.

The following are the mechanisms for understanding and influencing complex systems:

• Connections (aka relationships) represent exchanges between agents and determine the cohesiveness of the system.
• Perspectives (aka differences) refer to the diversity of agents within the system and provide the “energy” the system needs to be dynamic.
• Boundaries (aka containers) are what define the scope of the system and help to hold its components together in a pattern.

Lessons learned:

Three lessons we have learned about working with complex systems include:

• Ask a different kind of evaluation question. First, what is the system and what are its patterns? The mechanisms can help answer. Next, what patterns are wanted or needed? Third, how do we get there? Manipulate the mechanisms. Ask: Are we doing the right thing? Instead of: Are we doing things right?
• Quick and useful feedback. The evaluation questions are not answered just at the end of a project, they are ongoing explorations of a system that is always adapting and changing.
• Adapt as needed. Complex systems are adaptive, so don’t be afraid to adapt your evaluation methods, tools, or plans based on your observations and understandings of the system.

Rad Resources:  

• Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. (2010) Michael Patton. (Evaluation).
• Getting to Maybe: How the World is Changed. (2007). Frances Westley, Brenda Zimmerman, and Michael Patton. (Social change)
• Evaluating Systems Change: A Planning Guide. April 2010. Margaret B. Hargreaves. (Evaluation)

• Website: Human Systems Dynamics Institute (HSDI).

The American Evaluation Association is celebrating with our colleagues from Wilder Research this week. Wilder is a leading research and evaluation firm based in St. Paul, MN, a twin city for AEA’s Annual Conference, Evaluation 2012. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Systems

Hello, we are Rashon Lane, Alberta Mirambeau from the Centers for Disease Control and Prevention, and Steve Sullivan from Cloudburst Consulting and we work together on an evaluation aimed at assessing the uptake, use and impact of national public health hypertension recommendations by the Institute of Medicine (IOM). If you’ve ever wondered how to assess if public health programs are shifting their priorities to address evidence-based recommendations you might consider a methodology we used entitled alignment scoring analysis. In short, an alignment scoring analysis is a type of content analysis wherein narrative descriptions of organizational activities are analyzed to determine whether they support specific goals or strategies.  We conducted a pre-post alignment scoring analysis of state health department work plans to objectively determine if their project portfolios align with nationally recommended priorities.

Lessons Learned:

• Conduct pre-post content analysis. During our content analysis we coded state work plan activities as aligned, mis-aligned or neutral to the IOM recommendations.  As a result, we were able to share with program stakeholders that many state health departments were able to adjust their prevention priorities within 18 months to reflect national priorities.  If you are working on an evaluation to assess changes in priorities over time, you might consider conducting a similar pre-post content analysis to determine the degree to which public health programs align with priorities and how these priorities change over time.
• Use stringent criteria. Use stringent criteria to consider activities as aligned, mis-aligned or neutral for more accurate coding.

Hot Tips:

• Use a database. Use a database to facilitate the review of documents being analyzed and to speed reporting.  If you plan to use multiple reviewers, be sure to keep track of which reviewer coded a document so you can check inter-rater reliability and improve training on your coding protocol.
• Use alignment scoring. Use alignment scoring analysis results to provide recommendations to program stakeholders on how they might shift priorities that are NOT aligned with national recommendations that have proven to be effective.

Resources:

• Learn more about our evaluation in the article Epub ahead of print: Applying the Interactive Systems Framework to the Dissemination and Adoption of National and State Recommendations for Hypertension
• Learn more about alignment theory in Venkatraman’s seminal 1989 article:  The Concept of Fit in Strategy Research: Toward Verbal and Statistical Correspondence

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

alignment scoring analysis · content analysis · criteria · Database · evidence-based