AEA365 | A Tip-a-Day by and for Evaluators

TAG | Visualization

Hi, I’m Rebecca Woodland, an Associate Professor of Educational Leadership at UMass Amherst. If there is one thing that I know for certain it’s that relationships matter and how we are connected influences the quality and outcomes of our shared endeavors. Social Network Analysis (SNA) has had a profound influence on my evaluation work. I want to introduce and encourage evaluators (who may not know much about SNA) to consider integrating it into their own practice.

Simply put, SNA is all about telling the story of how “ties” between people or groups form, and how these “links” may influence important program objectives and outcomes. With SNA you can mathematically describe and visually see connections between people. You can use SNA to explain and predict how ties between “actors” influence the attainment of program goals.

Hot Tips: Evaluators can use SNA to address a wide-range of pressing program evaluation questions such as these:

  1. Want to know whether a program has the capacity to spread a new or novel intervention? SNA was used to evaluate school-level capacity to support or constrain instructional innovation.
  2. Want to know how large, inter-agency partnerships develop and how inter-agency collaboration correlates with intended program outcomes? Evaluators used SNA to track the development and impact of a Safe Schools/Healthy Students inter-agency community mental health network.
  3. Want to know who influences the budgeting and disbursement of funds for advocacy programs in fragile environments? SNA was used to map the flow of resources and funding patterns for new-born survival activities in northern Nigeria.

Lesson Learned: Possibly the biggest wow factor is that SNA enables the creation of illustrative visuals that display complex information, such as intra-organizational communication flow and the location of network “brokers,” “hubs,” “isolates” and “cliques”, in user-friendly ways.

WoodlandImage via under Creative Commons Attribution 3.0 License

Rad Resources

  • ®Visualyzer is an easy to use program (with a 30-day free trial) that enables you to create socio-grams on any network of interest to you.

The American Evaluation Association is celebrating Social Network Analysis Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello, we are William Faulkner (i2i Institute) and João Martinho (PlanPP), writing here on our own poster design process, which apparently worked well enough to impress some of the judges at AEA 2014. We were guided by a simple principle: understand what the target audience considers relevant and where this overlaps with that which we desire to communicate.

Faulkner 1

 

(Here’s a larger pdf of this poster: Network Analysis on a Shoestring_AEA2014)

We organized the content in three blocks:

  1. Orientation: what are we talking about and for whom is it relevant?
    • Who are you? The target audience – for whom we thought the content would be useful – because poster content is never relevant for everyone.
    • How do you collect data? We wanted to at least orient the audience to the range of types of data which could be fed into this tool.
    • Why would you use this tool? This box attempts to correct two common misconceptions: (a) that network analysis is only useful to map relationships between people, and (b) that producing a network visualization is the end of the process. The latter misconception inspires complaints that network analysts often produce attractive visualizations with little to no interesting interpretations.
  2. Main Message: what are the basic steps of using this tool? This block leads the reader through a tutorial on the main steps of using NodeXL emphasizing simplicity – in four steps NodeXL transforms raw data into a visualization. The section should display sufficient information to a solitary reader, but during the poster session itself at AEA we had one of the authors present with a laptop so anyone interested could play with a real dataset themselves as a way of reducing some of the mental entry barriers to starting to use the software.
  3. Examples/inspiration: The final block presents some concrete examples which illustrate the insights which network visualization (alone – even without the calculation of statistics) can supply.

Faulkner 2

Hot Tip: Focus on content first. The choice of a design tool should come after you can clearly articulate what you want to communicate and how this information is relevant to the target audience. Think about the gap you are trying to fill in the readers’ mind, and research how others communicated similar content. Second, as the design comes together, be strict about following the standard bank of recommendations about visual communication (less text, leave empty space, help the reader with cues about where their eye should go next). Once you have thoroughly thought through these aspects, the design should pretty much draw itself.

Rad Resource: NodeXL, of course! https://nodexl.codeplex.com/

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Feb/10

26

John LaVelle on Describing Evaluation

My name is John LaVelle.  One of the things that I have struggled with is describing what evaluation is and why it’s important.  Following up on that, many people ask about how evaluation is related to research.

Rad resource: Eva the Evaluator by Roger Miranda and Birgit Stadler.  This is a delightful and inexpensive book about a little girl named Eva and her father the evaluator.  Eva isn’t exactly sure what he father does, and so the book is about her father explaining evaluation in easily understandable terms, and Eva imagining herself engaged in the scenarios he described.  And, reading the book, I was pleasantly surprised with all the inside jokes, such as a cookbook titled “Preskill’s Chocolate Chip Cookies” and various evaluation theorists costumed as superheroes.

Hot Tip: Visualize the relationship like an hourglass.  I tend to be a visual person, and so I developed a picture of an hourglass with research on one side and evaluation on the other.  This picture illustrates how evaluation and research begin at different points and funnel to using the same methods to answer questions and analyze data.  They diverge again when it comes time to report on the outcomes (or processes) under study.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

My name is Jara Dean-Coffey, and I am the principal and founder of jdcPartnerships (www.jdcpartnerships.com). We partner with our clients to build their adaptive, strategic and leadership capacity and do so by using an evaluative inquiry approach to strategy formation, evaluation and assessment, leadership development and technical assistance and training. Our clients are in all sectors with the unifying purpose of striving to affect positive social change. Because we work in the intersection of disciplines, a lot of our initial work is about helping our clients get clear.

Hot Tip: Developing and using group process and facilitation skills is a important part of our tool kit and we are constantly finding ways to enhance. Often when working with a full organization, across organizations or community wide, we use a graphic recorder. A graphic recorder is, in short, a visual practitioner. The use of color, spatial placement of words, images and concepts has been incredibly powerful in our work around strategy and evaluation. The bonus: Clients LOVE IT. Instead of getting a 20 page bulleted report of a retreat, we give them a graphic record reflecting their words and process and a cover memo summarizing themes, recommendations and next steps. And for those who could not be at the meeting, they get a sense of the flavor of the conversation and its spirit and soul. Because ultimately, we are people doing this work. If you’d like to see an example created for our firm during a 2-day retreat to develop a Theory of Change with the full staff of a mid-size social sector organization, I have uploaded one into the AEA eLibrary at http://bit.ly/jdcGraphicFacilitationExample.

Rad Resource: Check out the International Forum of Visual Practitioners Website (http://www.visualpractitioner.org/) for more information on the history and range of skill that they can bring to a process. Another key resource is Grove Consultants (www.grove.com). They not only provide graphic facilitation services but a great range of tools that you can integrate in to your practice. They also have an affiliate program where you can find a consultants to support your work.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the
American Evaluation Association
. Please consider contributing – send a note of interest to aea365@eval.org.

· ·

My name is Laura Blasi.  I work in the field of assessment and institutional research, evaluate grants in the field of education, and teach research methods to graduate students.  In all cases I am trying to communicate in ways that are engaging but are still have meaning (to them and for me.)  I write a lot of reports, and I always want to convey my findings graphically, and I try to find a way to integrate the images into my writing.  My resource is using Sparklines to help convey visual data.

Rad Resource:  Edward Tufte in his book Beautiful Evidence defined sparklines as “intense, simple, wordlike graphics” – so when you see a graph illustrating the rise and fall of the dollar over several years and that graph is embedded in a paragraph only as high as the letters in this sentence.

For example:

Using sparklines I can show a trend and you do not have to break from reading to find a separate chart or graph somewhere else on the page.  About sparklines and more examples from outside of evaluation here: http://bit.ly/TufteSparklines

I wanted to share this here and would love to see other examples actually made in evaluation reports in our field – maybe knowing the importance and/or the impact would be great, too.  Has anyone done this for a funder or agency?

This contribution is from the aea365 Daily Tips blog, by and for evaluations, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

· ·

Archives

To top