Evaluation’s Role in a Data-Driven Reopening of Society by Elizabeth DiLuzio

Happy Saturday! I’m Elizabeth DiLuzio, AEA365 Curator + Contributor and Evaluation + Strategy Manager at Good Shepherd Services. Like many of you, I have been giving a lot of thought to our role as evaluator during COVID. More specifically, as our country shifts from stay-at-home initiatives to restore and reconstruct, how can we assist governments, schools, foundations, and organizations as they make the difficult decisions that lie ahead? 

The good news for us is that we have a lot to offer during this time. In fact, (most of) our political leaders have set the precedent that we use data to drive our next steps. So, today, I want to explore four key characteristics of a data-driven initiative and the related questions that we as data leaders need to be posing in order to move forward effectively.

Goal Oriented

Like any good initiative, it’s important to begin by identifying the organization’s goals and anti-goals in the coming months. Instead of the HiPPO (Highest Paid Person’s Opinion) making these decisions, it’s critical to convene large groups of diverse people and ask the questions in order to find our answers.

Questions to Ask: What is it that we want to accomplish? What are our goals? Our anti-goals?

Inquisitive

It’s easy to fall into the trap of basing our decisions on the data that’s available. Instead, it’s important to start from an inquisitive place. Christian Rudder, author of Dataclysm, speaks truth when he states that “behind every number there’s a person making decisions: what to analyze, what to exclude, what frame to set around whatever pictures the number paint. To make a statement, even to make just a simple graph, is to make choices, and in those choices human imperfection inevitably comes through.”

This is why, once goals are identified, we approach the next phase from a place of inquiry instead of a place of instant discernment. Here, too, we need to convene large groups of constituents to decide how we will measure the success of our goals (and what it looks like when we have reached our anti-goals).

Questions to Ask: What do we need to know in order to make this decision? What will the signals be when we are endangering our goals?

Transparent

At a minimum, this aspect of a data-driven culture entails openness about the aggregate data that is impacting leadership’s decisions. This means regular and diverse communication with constituents. At its most mature, this characteristic calls leaders to share the deidentified raw data they are using to make their decisions via an open data platform so that any constituent can do their own analysis at any time. This is a continuation of the spirit behind creating participatory goals and metrics – if people have access to their own data, they can not only be informed but help to make informed decisions in the future.

Questions to Ask: What channels of communication do we have available to us? What data are we willing and able to share?

Iterative

Even with the most brilliant minds working within the most perfect data-driven culture, the decisions that are made will not be perfect. There will inevitably be mistakes in the modeling, in the interpretation of the data, and in the decision-making. What matters is how you coach leadership through the response to these mistakes. To respond with an iterative, learning mindset is key here. This is where data-driven cultures are created and strengthened. 

Questions to Ask: What is possible and palatable right now?

Rad Resources

Curious to learn more? Check out Carl Anderson’s Creating a Data Driven Organization and DJ Patil + Hilary Mason’s Data Driven: Creating a Data Culture.

Now it’s your turn: What are some other important considerations we need to take into account as evaluators in the next phase of COVID? Share with us in the comments below or in our Evaluators’ Slack Channel, where you can comment, share links, and even upload resources. It’s easy to join and free to use. We’ll see you there!


Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

2 thoughts on “Evaluation’s Role in a Data-Driven Reopening of Society by Elizabeth DiLuzio”

  1. Hi Elizabeth,
    Great post! As a data scientist, I too am definitely thinking about how governments will use/rely on data to help us shift back to some sort of normalcy while we all deal with Covid. I am currently pursuing a Masters program in Education and one of the courses i am taking is on program evaluation. I’ve learned quite a bit about the role of an evaluator and more so what it takes to conduct a successful evaluation. Reading your post made it very clear that evaluation is very much dependent on data and successful evaluation requires a thorough understanding and analysis of available data. The Covid pandemic has led the government to rely heavily on data to help make decisions on next steps.

    I also want to mention that i like the four key characteristics you identified for data-driven initiatives that are important from an evaluation perspective. The characteristics that stood out to me the most was Transparency. I am big believer that the most successful evaluations are ones that use a collaborative/participatory approach. As you mentioned, having the raw data available on an open platform provides access to all involved in the evaluation process and it can help facilitate a participatory approach. I definitely think its important to not only have an understanding of data but also to understand that impact the data can have in helping promote data-driven decision making.

    Thank you!

  2. Dr. Sondra LoRe

    Great post-Elizabeth! I agree. The time is ripe to ask about what data can we collect and share to give voices to people who are meant to be served by programs. I really like the questions you have added throughout your post. Thank you!

    Sondra LoRe, Ph.D.
    Manager | National Institute for STEM Evaluation and Research (NISER)
    Adjunct Professor | Evaluation, Statistics, and Measurement Program, Department of Educational Psychology & Counseling
    The University of Tennessee, Knoxville
    Office of Research & Engagement
    114 Philander P. Claxton Education Building
    PH: 865-974-4962
    slore@utk.edu

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.