AEA365 | A Tip-a-Day by and for Evaluators

TAG | Systems thinking

We are Natalie Wilkins and Shakiyla Smith from the National Center for Injury Prevention and Control, Centers for Disease Control and Prevention.

As public health scientists and evaluators, we are charged with achieving and measuring community and population level impact in injury and violence prevention. The public health model includes: (1) defining the problem, (2) identifying risk and protective factors, (3) developing and testing prevention strategies, and (4) ensuring widespread adoption. Steps 3 and 4 have proven to be particularly difficult to actualize in “real world” contexts. Interventions most likely to result in community level impact are often difficult to evaluate, replicate, and scale up in other communities and populations.[i]

A systems framework for injury and violence prevention supplements the public health model by framing injury within the community/societal context in which it occurs.[ii] Communities are complex systems- constantly changing, self-organizing, adaptive, and evolving.  Thus, public health approaches to injury and violence prevention must focus more on changing systems, versus developing and testing isolated programs and interventions, and must build the capacity of communities to implement, evaluate, and sustain these changes.[iii] However, scientists and evaluators face challenges when trying to encourage, apply, and evaluate such approaches, particularly in collaboration with other stakeholders who may have conflicting perspectives. A systems framework requires new methods of discovery, collaboration, and facilitation that effectively support this type of work.

Lessons Learned:

  • Evaluators can use engagement and facilitation skills to help stakeholders identify their ultimate goals/outcomes and identify the systems within which these outcomes are nested (Goodman and Karash’s Six Steps to Thinking Systemically provides an overview for facilitating systems thinking processes).
  • Evaluators must also address and communicate around high-stakes, conflictual issues that often undergird intractable community problems. “Conversational capacity”[iv] is an example of a skillset that enables stakeholders to be both candid and receptive in their interactions around challenging systems issues.

Rad Resources:

  • Finding Leverage: This video by Chris Soderquist provides an introduction to systems thinking and how it can be applied to solve complex problems.
  • The Systems Thinker: Includes articles, case studies, guides, blogs, webinars and quick reference “pocket guides” on systems thinking.

i Schorr, L., & Farrow, F. (2014, November). An evidence framework to improve results. In Harold Richman Public Policy Symposium, Washington, DC, Center for the Study of Social Policy.

ii McClure, R. J., Mack, K., Wilkins, N., & Davey, T. M. (2015). Injury prevention as social change. Injury prevention, injuryprev-2015.

iiiSchorr, L., & Farrow, F. (2014, November). An evidence framework to improve results. In Harold Richman Public Policy Symposium, Washington, DC, Center for the Study of Social Policy.

iv. Weber, C. (2013) Conversational capacity: The secret to building successful teams that perform when the pressure is on.  McGraw Hill Education: New York, NY

The American Evaluation Association is celebrating Community Psychology (CP) TIG Week with our colleagues in the CP AEA Topical Interest Group. The contributions all this week to aea365 come from our CPTIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

· ·

Hi! We are Sara Vaca (EvalQuality.com) and Pablo Vidueira (professor at the Universidad Politécnica de Madrid). Today we are going to talk about the benefits of using the latest advances in data visualization to improve ST tools.

Systems Thinking (ST) is the new paradigm in Evaluation which represents a significant mind-set shift and a powerful tool to tackle complex environments. It refers to the adoption of concepts, methodologies and tools coming from the systems field.

Lesson Learned: ST already use data visualizationVaca 1

Among the wealth of tools, concepts and approaches within the systems field there are hard and soft systems approaches. Among soft systems, rich pictures and the soft systems methodology are widely used. In the hard systems side, system dynamics (SD) is one of the most famous systems approaches.

And all these tools already use data visualization: they depict ideas, relationships and concepts relying in shapes and figures more than a textual explanation.

Rad Resource: Knowing how graphical perception works

For many years vision researchers have been investigating how the human visual system analyses images. An important initial result was the discovery of a limited set of visual properties that are detected very rapidly and accurately by the low-level visual system.

An important discovery of early studies investigating how the human visual system analyzes images was the identification of a limited set of visual features that are detected very rapidly by low-level, fast-acting visual processes. These properties were initially called preattentive, since their detection seemed to precede focused attention, occurring within the brief period of a single fixation. Attention plays a critical role in what we see, even at this early stage of vision. The most relevant pre-attentive visual features are: orientation, length, width, closure, size, curvature, density, contrast, number, estimation and color.

Cool Trick: Using graphical perception principles to improve the ST tools Vaca 2

 

 

Vaca 3

 

 

We are studying ST tools conventions of symbols and are working on variations to broaden its variety using simple features. For example: in the typical standard scheme for a Stock and Flow diagram, we are playing with the width of the arrows to represent the relevance of each variable. Thicker arrows (flow variable 2 and auxiliary A) would indicate bigger influence than thinner arrows (flow variable 3 and auxiliary B)

 

Another example would be replacing +/- symbols in causal loop diagrams by colors (green=positive, red=negative), to make the causal relationships between variables easier to interpret.

We think these improvements would make these tools more informative for those using them and more attractive for those new to them.

We welcome your reactions and hope to share an upcoming paper on this topic with you in Chicago!Vaca 4

 

 

 

Vaca 5

 

 

 

The American Evaluation Association is celebrating Research on Evaluation (ROE) Topical Interest Group Week. The contributions all this week to aea365 come from our ROE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

I’m Kylie Hutchinson, independent evaluation consultant and trainer with Community Solutions Planning & Evaluation. I also tweet regularly at @EvaluationMaven.

Systems thinking and evaluation is a hot topic these days, and as someone who spends a fair bit of time in evaluation capacity building, it has me thinking a lot about logic models. Some of you might recall a post I did for AEA365 back in 2014 on “Are Logic Models Passé?’, where I mused about the utility of static logic models in highly dynamic and complex programs.

Since then I’ve been on the hunt for examples of more “fuzzy” logic models but have only been able to find one example. Which leads me to wonder if what programs really need is not something “fuzzy”, but rather something that is both structured and flexible at the same time. Sort of like Lego.Lego Bridge #2

Imagine that a program is a bridge, designed to get people from one side of a canyon to another. The program logic model is the bridge’s design, and the better the design, the greater the chances of receiving funding to build it. The bridge construction initially occurs according to plan, however, as time goes on things come up and the bridge contractor wants to make some changes. What do you do? Stick with the original design but risk not reaching the other side? If your bridge is made of steel or concrete, you’re stuck moving forward. But if you build it with Lego, it’s easier to swap pieces in and out, without having to demolish the whole bridge. Eventually you’ll get to the other side, but maybe the bridge looks a bit different than you originally intended.

I know that some funders and government departments aren’t comfortable with the idea of “fuzzy” and I can appreciate that. Perhaps a Lego bridge is something more in line with their needs.

Rad Resource: Here are two Pinterest pages with resources on both logic models and systems evaluation.

Rad Resource: For a quick overview of systems thinking and evaluation, check out this five minute video.

The American Evaluation Association is celebrating Logic Model Week. The contributions all this week to aea365 come from evaluators who have used logic models in their practice. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

We are Debra Smith and Galen Ellis, two evaluators who discovered through AEA that we share a common method of using logic models to facilitate systems thinking with our clients. Many people think logic models are a complicated exercise with little value. Some are downright cynical, saying they tend to represent “a tenuous chain of unproven assumptions used to justify the pre-determined program model” (Public Health Director).

We both use a two-phase logic model development process: first, we help our clients develop a balcony view “theory of change” by identifying the global goal or vision and mapping key resources, strategies and outcomes. Clarity in Phase I, makes going to Phase II—identifying outputs and short, mid and long-term outcomes and measures—more manageable and meaningful.

Debra: I first used this approach while working with a museum education department to develop an evaluation system for their programs. We mapped the overall theory of the department, tracking resources and activities leading to their long-term vision, which they described as “the community loving the museum.” Staff were then able to develop logic models for their individual programs, and then a system that streamlined the data they collected within and across programs.

Galen: I have facilitated logic model processes for the development of agency-wide evaluation systems with several organizations in this two-step process. The theory of change process helps the client articulate how their activities and the outcomes they expect fit with their agency’s values and mission. Then I work with each individual program/project within the organization to develop its own logic models that link to the agency’s broader theory of change. This shifts the culture of the organization towards being outcomes-based, and helps connect the distinct programs via common outcomes that reflect the agency’s values and mission.

Lessons Learned:

  • Logic models can help prevent mission drift. The agency-level logic model will capture outcomes that are aligned with the mission. Programs within the organization can then align with those outcomes and share evaluation measures, leveraging the broader organizational goals to guide their own success.
  • Using the logic model process to develop an agency-wide evaluation system elevates the value of evaluation within the organization.

Rad Resources:

Hot Tips:

  • Showing how a logic model tells a story can help clients understand the role and value of a logic model. Galen uses the metaphor of crossing a river. Video Clip
  • Even in developmental projects, it can be helpful to map the theory of change, then refine it based on what is learned.

The American Evaluation Association is celebrating Logic Model Week. The contributions all this week to aea365 come from evaluators who have used logic models in their practice. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hi, I’m Nora F. Murphy, a developmental evaluator and co-founder of TerraLuna Collaborative. Qualitative Methods have been a critical component of every developmental evaluation I have been a part of. Over the years I’ve learned a few tricks about making qualitative methods work in a developmental evaluation context.

Hot Tip: Apply systems thinking. When using developmental evaluation to support systems change it’s important to apply systems thinking. When thinking about the evaluation’s design and methods I am always asking: Where are we drawing the boundaries in this system? Whose perspectives are we seeking to understand? What are the important inter-relationships to explain? And who benefits or is excluded by the methods that I choose? Qualitative methods can be time and resource intensive and we can’t understand everything about systems change. But it’s important, from a methodological and ethical perspective to be intentional about where we draw the boundaries, whose perspectives we include, and which inter-relationships we explore.  

Hot TipPractice flexible budgeting. I typically budget for qualitative inquiry but create the space to negotiate the details of that inquiry. In one project I budgeted for qualitative inquiry that would commence six months after the contract was finalized. It was too early to know how strategy would develop and what qualitative method would best for learning about the developing strategy. In the end we applied systems thinking and conducted case studies that looked at the developing strategy in three ways: from the perspective of individual educators’ transformation, from the perspective educators participating in school change, and from the perspective of school leaders leading school change. It would have been impossible to predict that this was the right inquiry for the project at the time the budget was developed.

Hot Tip: Think in layers. The pace of developmental evaluations can be quick and there is a need for timely data and spotting patterns as they emerge. But often there is a need for a deeper look at what is developing using a method that takes more time. So I think in layers. With the case studies, for example, we structured the post-interview memos so they can be used with program developer to spot emergent patterns by framing memos around pattern surfacing questions such as: “I was surprised…  A new concept for me was… This reinforced for me… I’m wondering…” The second layer was sharing individual case studies. The third layer was the cross-analysis that surfaced deeper themes. Throughout we engaged various groups of stakeholders in the meaning making and pattern spotting.

Rad Resources:

The American Evaluation Association is celebrating Developmental Evaluation Week. The contributions all this week to aea365 come from evaluators who do developmental evaluation. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

I’m Katrina Brewsaugh, Senior Associate with the Annie E. Casey Foundation.  As part of Social Work TIG week on aea365, I’d like to tell you why social workers are an asset to an evaluation team.  Evaluation may not be the first thing that comes to mind when you think of social work, but the training we receive can be an asset to evaluation teams in several ways.

Lesson Learned:  Social workers are systems thinkers.

Ecological systems theory is at the core of social work training, from the bachelor’s degree through the doctorate.  A social worker on your team can identify issues and concerns that impact the micro- (family), mezzo- (community), and macro- (policy) levels, regardless of whether the evaluation is localized to one small program or has a national policy reach.

Lesson Learned:  Social workers are interdisciplinary.

Social work synthesizes knowledge of related fields such as psychology, sociology, education, public health, and even economics.  On interdisciplinary evaluation teams, a social worker can be an excellent translator and present a unified voice.

Lesson Learned:  Social workers are hands on.

All social workers must complete training in the field working with clients.  If you need someone skilled in engaging and interviewing consumers and stakeholders, then you need a social worker!

Rad Resources:  Here are some places to begin your search for a social work evaluator:

  • Search AEA’s Find an Evaluator list using ‘social work’ or ‘MSW’ as a search term.
  • Contact your local university’s social work program.
  • Contact a member of the Social Work TIG (through the “members only” link on www.eval.org).
  • Look for social workers on EvalTalk or AEA’s LinkedIn page.

So the next time you need to bring together an evaluation team, consider a social worker!

The American Evaluation Association is celebrating SW TIG Week with our colleagues in the Social Work Topical Interest Group. The contributions all this week to aea365 come from our SW TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi! My name is Natalia Kosheleva. I’m a consultant at Process Consulting Company, Moscow, Russia. Here I would like to share my ideas about measuring the extent of program intervention into targeted systems.

I base my work on two systems thinking principles. The Hierarchy principle says that any system is an element of some larger system, and any element is a system itself. The Multiple Description principle says that no single model can give an exhaustive description of a system, so several models should be developed.

I start with describing a program as a system that includes a source/sources of intervention and a target object/objects. For example, there is an NGO (source) working with schools (objects) to promote student community service.

Then I go through several levels of hierarchy to get a detailed description of targeted system(s), always getting to the level of individual “functions”. For example, a school has administrators, teachers and students, and students are organized into classes. At this stage it is helpful to do a lot of sketches.

Hot Tip: The effect of intervention depends upon what type of elements it affects and what share of elements of a certain type gets affected within a target system. So it is useful to track that. For example, a program might have trained 20 8th grade students from two schools, 10 from each. But one school has just one class of 18 people, so the intervention affected 100% of target classes and 56% of students. Another school has three classes of 30 students each, and out of 10 trained students 2 were from one class and 8 from another. So the extent of program intervention here is 67% of classes and 7, 27, and 0% of students in these classes.

Hot Tip: The effect of intervention also depends upon how deeply a program influences individuals. Many social programs try to create change through training. For them it may be useful to use a qualitative scale, like the one I used for a program training students on community service (CS): “Received training in CS”, “Received training and practiced CS”, etc. The scale, of course, should be program-specific.

A qualitative scale can be combined with quantitative indicators above. For example:

System/ Type of Element Total number of elements Received training 

Number/ Share

Received training and practiced doing 

Number/ Share

School 1/ Class 1/ Students 20 10/ 56% 1/ 5%
School 2/ Class 1/ Students 30 2/ 7% 2/ 7%
School 1/ Class 2/ Students 30 8/ 26% 4/ 13%

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

Archives

To top