AEA365 | A Tip-a-Day by and for Evaluators

TAG | Systems Concepts

Hi! We are Sara Vaca ( and Pablo Vidueira (professor at the Universidad Politécnica de Madrid). Today we are going to talk about the benefits of using the latest advances in data visualization to improve ST tools.

Systems Thinking (ST) is the new paradigm in Evaluation which represents a significant mind-set shift and a powerful tool to tackle complex environments. It refers to the adoption of concepts, methodologies and tools coming from the systems field.

Lesson Learned: ST already use data visualizationVaca 1

Among the wealth of tools, concepts and approaches within the systems field there are hard and soft systems approaches. Among soft systems, rich pictures and the soft systems methodology are widely used. In the hard systems side, system dynamics (SD) is one of the most famous systems approaches.

And all these tools already use data visualization: they depict ideas, relationships and concepts relying in shapes and figures more than a textual explanation.

Rad Resource: Knowing how graphical perception works

For many years vision researchers have been investigating how the human visual system analyses images. An important initial result was the discovery of a limited set of visual properties that are detected very rapidly and accurately by the low-level visual system.

An important discovery of early studies investigating how the human visual system analyzes images was the identification of a limited set of visual features that are detected very rapidly by low-level, fast-acting visual processes. These properties were initially called preattentive, since their detection seemed to precede focused attention, occurring within the brief period of a single fixation. Attention plays a critical role in what we see, even at this early stage of vision. The most relevant pre-attentive visual features are: orientation, length, width, closure, size, curvature, density, contrast, number, estimation and color.

Cool Trick: Using graphical perception principles to improve the ST tools Vaca 2



Vaca 3



We are studying ST tools conventions of symbols and are working on variations to broaden its variety using simple features. For example: in the typical standard scheme for a Stock and Flow diagram, we are playing with the width of the arrows to represent the relevance of each variable. Thicker arrows (flow variable 2 and auxiliary A) would indicate bigger influence than thinner arrows (flow variable 3 and auxiliary B)


Another example would be replacing +/- symbols in causal loop diagrams by colors (green=positive, red=negative), to make the causal relationships between variables easier to interpret.

We think these improvements would make these tools more informative for those using them and more attractive for those new to them.

We welcome your reactions and hope to share an upcoming paper on this topic with you in Chicago!Vaca 4




Vaca 5




The American Evaluation Association is celebrating Research on Evaluation (ROE) Topical Interest Group Week. The contributions all this week to aea365 come from our ROE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.


Kia Ora! I’m Bob Williams. In our book Systems Concepts in Action : A Practitioner’s Toolkit, Richard Hummelbrunner and I distinguished between describing situations, thinking systemically, and being systemic.  I’ve see these notions describing three stages of a journey.  As you read these three scenario, pose yourself the following questions.  How well do the scenario describe my own journey?  In what way do the similarities and differences matter?  Who or what can help me move further along my journey?

Describing situations (or systems).  During this part of the journey you may be talking about systems as ‘real’ things, often big things (eg. the health system or the school system).  You have acknowledged that much of what you observe and describe is complex.  You may have heard about holism and trying to include everything into your evaluations.  You are seeing how inter-relationships create observable and significant patterns.  You are describing fresh differences that make a difference.  On the other hand you may feel overwhelmed by the sheer scale of what you need to consider.  You are starting to be worried about practicality and how to simplify in order to get your head around the vastness of it all.

Thinking systemically.  At this point in your journey you may be simplifying by considering ‘systems’ less as real life entities and more as mental models that help you think about ‘situations’.  You are engaging in how different people ‘see’ the same situation in entirely different ways and learning more ways to set boundaries around your systemic thinking.  You are probably looking at specific systems and complexity methods in order to help you with this process.  You are applying some of these approaches and gaining deeper insights into how to evaluate messy situations.  On the other hand, you may be frustrated by the range of methods and uncertain which ones work best in which circumstances.

Being systemic.  You find that you intuitively understand inter-relationships, engage with multiple perspectives and reflect deeply on the practical and ethical consequences of the boundary choices you make.  You use these insights with existing evaluation approaches rely less on specific systems methods.  You probably realise that choosing values that underpin your judgments of merit, worth and significance is a form of boundary setting.

Hot Tip: Every endeavour is bounded.  We cannot do or see everything.  Every viewpoint is partial.  Therefore, holism is not about trying to deal with everything, but being methodical, informed, pragmatic and ethical about what to leave out.  And, it’s about taking responsibility for those decisions.

Bob Williams received the 2014 AEA Lazarsfeld Award for contributions to “fruitful debates on the assumptions, goals and practices of evaluation.”

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.


My name is Beverly Parsons. I’m the executive director of InSites (a non-profit research, evaluation, and planning organization) and an AEA board member. I have a tip about resources for applying systems concepts to evaluation. This tip is based on InSites’ continually expanding use of systems orientations in evaluations in education, social services, community change, and health.

Hot Tip: Whether you are an old hand at using systems concepts or new to the topic, get your hands on a copy of Ramage and Shipp’s 2009 book, Systems Thinkers (Springer London). This book is about the people who have shaped the systems fields across many disciplines. The book’s authors organize systems thinkers around themes—cybernetics, general systems theory, systems dynamics, complexity theory, soft and critical systems, and learning systems. Ramage and Shipp’s focus on the person behind the idea helps you see how differing ways of thinking systemically are linked to people’s personalities, lives, environment, time in history, and connections with others.

Each of the book’s 30 chapters gives a brief, well written synopsis of a system thinker’s background, contribution, and links to other thinkers along with a short extract from his/her writings. You’ll be drawn to some because their ideas are familiar, to others because of their life experience or personality, and yet others because of their ideas are new to you. Explore the landscape of systems, walking beside these colorful characters from many disciplines.

Rad Resources: Just as Ramage and Shipp give us a way to position ourselves in the landscape of systems thinkers broadly, you can do the same with your contemporaries who are applying systems thinking to evaluation. Get to know the many people who are regular contributors to the application of systems thinking to evaluation. You can find them by attending AEA sessions sponsored by the Systems in Evaluation TIG. Go to one or both of the November pre-conference workshops (Systems Thinking and Evaluation Practice: Tools to Bridge the Gap on Tuesday or Useful Tools for Integrating System Dynamics and System Intervention Elements into System Change Evaluation Designs on Wednesday) where you’ll meet six workshop leaders who are each crafting their own ways of synthesizing and applying system concepts. These sessions will help you traverse the rest of the AEA program and find kindred spirits.

If you can’t attend AEA this year, check out Michael Quinn Patton’s new book, Developmental Evaluation, Jonny Morell’s new book, Evaluation in the Face of Uncertainty, the websites of Bob Williams ( and Glenda Eoyang ( as well as our website (, and the e-libraries of AEA and the Canadian Evaluation Society. You’ll find great resources, discover what you are already doing that is systems-based, identify areas you want to explore, and find people to walk along side in your journey of applying systems thinking to evaluation practice. Draw on both the relationships and the ideas. Happy trails.

The American Evaluation Association is celebrating Systems in Evaluation Week with our colleagues in the Systems in Evaluation AEA Topical Interest Group. The contributions all this week to aea365 come from our Systems TIG members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting Systems resources. You can also learn more from the Systems TIG via their many sessions at Evaluation 2010 this November in San Antonio.

· ·


To top