I’m Carlos Romero, Senior Evaluator at Apex, an evaluation firm in Albuquerque. New Mexico is rated the worst state to raise a family: 49th in poverty, 43rd in health and safety, and 49th in education among other indicators. It’s hard to digest this data, partly because I think New Mexico is an amazing place to live and because these are huge needles to move. But also because these data don’t reflect the progress we’re making. It’s a reminder how much context matters to give meaning to data. In evaluation and in life, understanding context helps us define problems accurately, formulate appropriate strategies, and establish realistic expectations as we measure progress toward change.
Consider our education ranking, which is based on the percentage of students who graduate high school on time. We improved from 60% in 2011 to 71% in 2017, bridging the gap to the national rate of 80% but not changing our dismal ranking. I’m honored to know and work with some of the people and programs that are contributing to this change. But it also made me think of a charter school in Albuquerque that helps young people who did not graduate on time earn a diploma instead of a GED. They were rated a failing school because the diploma is not “on time” and their success is not reflected in our national ranking. When you remove the “on time” distinction, 85% of New Mexicans have a high school diploma compared to 87% nationally. Why and how does “on time” matter? We need context.
Lesson Learned: Systems thinking can help reveal and understand context using distinction-making, part-whole systems structures, relationships, and perspectives. At Apex we use DSRP as a verb. Let’s DSRP education in a diagram.
- The percentage of people who graduate on time is a distinction and implies another group of people who graduated not on time.
- To better understand the big picture, let’s group education levels for the entire population. Why, then, do we care about education? ?
- Let’s look at other parts of our societal system: income and health. This zooming in and out is an example of part-whole systems structure.
- Higher education level is associated with higher income, which is associated with higher levels of health. These are established relationships.
- But is the relationship between graduating on time and graduating not on time important? That’s a matter of perspective. If you graduate on time, you are more likely to continue your education and earn more money. ?
- But whether you get it on time or not, from the perspective of money, a high school diploma is much more valuable than a GED, which is much better than dropping out altogether. Where do we set the bar for success?
Lessons learned: Don’t be discouraged or distracted by data without context.
Hot Tip: Use Systems Thinking “DSRP” questions and diagramming to reveal and understand context. What are the important distinctions? What are the “whole and parts” of those distinctions? How are those distinctions and their parts related or not related? What relationships matter most? What are all the possible perspectives be they human or abstract such as the perspectives of time and money?
The American Evaluation Association is celebrating New Mexico (NM) Evaluators (www.nmeval.org) Week. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.