I’m Carlos Romero, Senior Evaluator at Apex, an evaluation firm in Albuquerque. New Mexico is rated the worst state to raise a family: 49th in poverty, 43rd in health and safety, and 49th in education among other indicators. It’s hard to digest this data, partly because I think New Mexico is an amazing place to live and because these are huge needles to move. But also because these data don’t reflect the progress we’re making. It’s a reminder how much context matters to give meaning to data. In evaluation and in life, understanding context helps us define problems accurately, formulate appropriate strategies, and establish realistic expectations as we measure progress toward change.
Consider our education ranking, which is based on the percentage of students who graduate high school on time. We improved from 60% in 2011 to 71% in 2017, bridging the gap to the national rate of 80% but not changing our dismal ranking. I’m honored to know and work with some of the people and programs that are contributing to this change. But it also made me think of a charter school in Albuquerque that helps young people who did not graduate on time earn a diploma instead of a GED. They were rated a failing school because the diploma is not “on time” and their success is not reflected in our national ranking. When you remove the “on time” distinction, 85% of New Mexicans have a high school diploma compared to 87% nationally. Why and how does “on time” matter? We need context.
Lesson Learned: Systems thinking can help reveal and understand context using distinction-making, part-whole systems structures, relationships, and perspectives. At Apex we use DSRP as a verb. Let’s DSRP education in a diagram.
- The percentage of people who graduate on time is a distinction and implies another group of people who graduated not on time.
- To better understand the big picture, let’s group education levels for the entire population. Why, then, do we care about education? ?
- Let’s look at other parts of our societal system: income and health. This zooming in and out is an example of part-whole systems structure.
- Higher education level is associated with higher income, which is associated with higher levels of health. These are established relationships.
- But is the relationship between graduating on time and graduating not on time important? That’s a matter of perspective. If you graduate on time, you are more likely to continue your education and earn more money. ?
- But whether you get it on time or not, from the perspective of money, a high school diploma is much more valuable than a GED, which is much better than dropping out altogether. Where do we set the bar for success?
Lessons learned: Don’t be discouraged or distracted by data without context.
Hot Tip: Use Systems Thinking “DSRP” questions and diagramming to reveal and understand context. What are the important distinctions? What are the “whole and parts” of those distinctions? How are those distinctions and their parts related or not related? What relationships matter most? What are all the possible perspectives be they human or abstract such as the perspectives of time and money?
The American Evaluation Association is celebrating New Mexico (NM) Evaluators (www.nmeval.org) Week. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hi Carlos,
I enjoyed reading your post, as I am guilty of sometimes looking at raw data without context and inevitably become misinformed and misguided because of it. I am a teacher at a private school in Vancouver, Canada, and before that I worked at a public school in a lower socioeconomic part of the city. Levels of success at these two schools are perceived very differently, and context is important when determining success. For example, at the private school they advertise that 99% of students attend college or university after graduation, however they don’t say how many people actually complete their post-secondary degree. From looking at that statistic, you’d assume that virtually everyone who graduates from my school ends up with a college or university degree, which is FAR from the truth (I literally ran into two of my old students yesterday who dropped out of college after one year).
I am currently completing my Masters in Education which has led me to program evaluation, and it is an important reminder that context is needed when obtaining data for evaluations. Simple numbers and graphs can be misleading if there is no context given, and we all need to keep this in mind when reading reports and implementing changes to our programs.
Thank you for your post, it’s given me a lot to think about!
Heather
Fascinating article. I will share it with Atrisco Heritage High School’s Community Council Board which I am on. I believe your article hits upon an issue relevant to their situation and one which we are working to correct.
Sincerely,
Scott Durán
Dear Carlos,
I’m currently taking my Masters in Education and found your article to be both extremely useful and deeply interesting in the context of a class I’m taking on evaluation. I found your DSRP strategy to be an excellent solution to questions I’ve been having about how to resolve the issue of context in evaluations where public data seems to be skewed or misrepresentative of a program evaluation. I was also wondering if you had any thoughts as to how this might function in a more qualitative situation – for example, in a program concerned with mental health, or with an issue that necessarily would incorporate both quantitative and qualitative data as a matter of course in an evaluation?
More specifically, I’m curious if you have any experience with diagraming a situation of this kind? As I said, I found your diagram extremely helpful to my understanding of the situation your article focuses on, and I am keen to see if I can apply a similar model in my class to a more mixed set of qualitative and quantitative data.
Additionally, on a somewhat unrelated note, I was very taken aback by the data you described. It is indeed true that New Mexico is regarded as a state very low down on the national rankings in several key areas, and whilst this may be a somewhat naïve question, I’m curious about your opinion on why data which clearly misrepresents – as you point out – the actual state of high school diploma achievement in New Mexico continues to be used by national authorities. I was also wondering if you have any suggestions of other resources that could help me understand more about why this data is used. Essentially, I am interested in whether or not there have been any clear arguments made about why this data relies so overwhelmingly on graduating ‘on-time’ without taking into account context which may mean this to be an erroneous focus.
Aside from this, thank you for your article! As I said I found it interesting and it has shed light on some issues in my class I was struggling to understand.
Lesley Machon