Happy Saturday! I’m Elizabeth DiLuzio, AEA365 Curator + Contributor and Evaluation + Strategy Manager at Good Shepherd Services. Like many of you, I have been giving a lot of thought to our role as evaluator during COVID. More specifically, as our country shifts from stay-at-home initiatives to restore and reconstruct, how can we assist governments, schools, foundations, and organizations as they make the difficult decisions that lie ahead?
The good news for us is that we have a lot to offer during this time. In fact, (most of) our political leaders have set the precedent that we use data to drive our next steps. So, today, I want to explore four key characteristics of a data-driven initiative and the related questions that we as data leaders need to be posing in order to move forward effectively.
Goal Oriented
Like any good initiative, it’s important to begin by identifying the organization’s goals and anti-goals in the coming months. Instead of the HiPPO (Highest Paid Person’s Opinion) making these decisions, it’s critical to convene large groups of diverse people and ask the questions in order to find our answers.
Questions to Ask: What is it that we want to accomplish? What are our goals? Our anti-goals?
Inquisitive
It’s easy to fall into the trap of basing our decisions on the data that’s available. Instead, it’s important to start from an inquisitive place. Christian Rudder, author of Dataclysm, speaks truth when he states that “behind every number there’s a person making decisions: what to analyze, what to exclude, what frame to set around whatever pictures the number paint. To make a statement, even to make just a simple graph, is to make choices, and in those choices human imperfection inevitably comes through.”
This is why, once goals are identified, we approach the next phase from a place of inquiry instead of a place of instant discernment. Here, too, we need to convene large groups of constituents to decide how we will measure the success of our goals (and what it looks like when we have reached our anti-goals).
Questions to Ask: What do we need to know in order to make this decision? What will the signals be when we are endangering our goals?
Transparent
At a minimum, this aspect of a data-driven culture entails openness about the aggregate data that is impacting leadership’s decisions. This means regular and diverse communication with constituents. At its most mature, this characteristic calls leaders to share the deidentified raw data they are using to make their decisions via an open data platform so that any constituent can do their own analysis at any time. This is a continuation of the spirit behind creating participatory goals and metrics – if people have access to their own data, they can not only be informed but help to make informed decisions in the future.
Questions to Ask: What channels of communication do we have available to us? What data are we willing and able to share?
Iterative
Even with the most brilliant minds working within the most perfect data-driven culture, the decisions that are made will not be perfect. There will inevitably be mistakes in the modeling, in the interpretation of the data, and in the decision-making. What matters is how you coach leadership through the response to these mistakes. To respond with an iterative, learning mindset is key here. This is where data-driven cultures are created and strengthened.
Questions to Ask: What is possible and palatable right now?
Rad Resources
Curious to learn more? Check out Carl Anderson’s Creating a Data Driven Organization and DJ Patil + Hilary Mason’s Data Driven: Creating a Data Culture.
Now it’s your turn: What are some other important considerations we need to take into account as evaluators in the next phase of COVID? Share with us in the comments below or in our Evaluators’ Slack Channel, where you can comment, share links, and even upload resources. It’s easy to join and free to use. We’ll see you there!
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hello Elizabeth,
I am reading your post a little over a year after you had written it and (sadly) it seems as relevant today as it did back then. COVID continues to effect the lives of everyone, and decisions regarding how to combat this disease and return to a sense of normalcy continue to be made.
The 4 key characteristics you identified for data-driven initiatives from an evaluation perspective continue to be important today, and yet politicians (depending where you live) do not necessarily abide by them. I live in Ontario, Canada, where clear goals have been set, data has been sought out, the process has certainly been iterative, but what I believe we are missing, is transparency. Politics continue to get in the way preventing an open and honest discussion about where we stand in terms of sources of viral transmission.
This missing link, transparency, continues to hold back progress by not allowing the public to hold decision makers to account. The best science can be performed to acquire high quality data, but if we cannot get our audience to listen to it then it is for naught. How then, do we improve transparency? How do we hold our decisions makers to account? We cannot wait another year until the next election to force a change in leadership. What role could evaluators fill to make the evaluation process more transparent?
I hope to hear your thoughts.
Chris
Thank you for your reflections, Laxshaa, and I’m happy to hear the blog resonated with you!
Hi Elizabeth,
Great post! As a data scientist, I too am definitely thinking about how governments will use/rely on data to help us shift back to some sort of normalcy while we all deal with Covid. I am currently pursuing a Masters program in Education and one of the courses i am taking is on program evaluation. I’ve learned quite a bit about the role of an evaluator and more so what it takes to conduct a successful evaluation. Reading your post made it very clear that evaluation is very much dependent on data and successful evaluation requires a thorough understanding and analysis of available data. The Covid pandemic has led the government to rely heavily on data to help make decisions on next steps.
I also want to mention that i like the four key characteristics you identified for data-driven initiatives that are important from an evaluation perspective. The characteristics that stood out to me the most was Transparency. I am big believer that the most successful evaluations are ones that use a collaborative/participatory approach. As you mentioned, having the raw data available on an open platform provides access to all involved in the evaluation process and it can help facilitate a participatory approach. I definitely think its important to not only have an understanding of data but also to understand that impact the data can have in helping promote data-driven decision making.
Thank you!
Great post-Elizabeth! I agree. The time is ripe to ask about what data can we collect and share to give voices to people who are meant to be served by programs. I really like the questions you have added throughout your post. Thank you!
Sondra LoRe, Ph.D.
Manager | National Institute for STEM Evaluation and Research (NISER)
Adjunct Professor | Evaluation, Statistics, and Measurement Program, Department of Educational Psychology & Counseling
The University of Tennessee, Knoxville
Office of Research & Engagement
114 Philander P. Claxton Education Building
PH: 865-974-4962
slore@utk.edu