How a Lean Data Approach Can Change Your Effectiveness as an Evaluator by Elizabeth DiLuzio

Hi everyone, Elizabeth DiLuzio here. I’m the newest AEA365 curator and Manager of Evaluation and Strategy at Good Shepherd Services in NYC. Today, I want to share a semi-new approach to impact evaluation called lean data.

The use case for a lean data approach is quite compelling: it’s a decision-making tool for when time and money are short and there is a pressing need to make rapid operational decisions. Who of us hasn’t been in that position before? The lean data approach can also help if you feel most of your time is spent satisfying funder requirements, or if the data collection process is so lengthy that any findings are irrelevant by the time you share them with program staff.

But what exactly is this seeming panacea? Let’s take a closer look.

What is it? 

Lean data is a survey approach to measuring and managing a program’s impact that was first piloted by Acumen in 2014. A lean data approach does not replace but rather complements your current impact evaluation method(s). It applies three lean experimentation principles

  • Customer First – goodbye compliance mindset, hello value for customers and program staff 
  • Low-Cost Technology – finding ways to harness technology to create rapid feedback loops
  • Data-Driven Decision Making – asking the essential questions that, when answered, can aid in deciding on next steps

How does it work?

Rather than taking a “what data do we need?” approach, lean data takes a “what am I trying to accomplish and what information would help me to get there?” approach. The whole process consists of 4 stages:

  1. Get Started. In this stage, you will identify the impact you are trying to achieve and what information you need in order to get there.
  2. Choose Your Technology. Here, you’ll want to choose the type of data collection technology that works the best for your population.
  3. Choose Your Questions and Method. When identifying or creating your questions, be sure that each is relevant to what you need to know, and that the response you receive will drive some type of action. When it comes to methods, you’ll want to map out the program’s user journey in order to pinpoint the times that staff are already in contact with your surveyees. Any of those moments, called touchpoints, are when your questions can be embedded into already existing operations. This is designed to lessen the burden on the clients and staff.
  4. Take Action. This might be the most important stage! The purpose of this approach is to inform a concrete decision, so be sure to take action once you get the information you need.

Rad Resource

Curious to learn more? Check out Acumen’s free online course for a more in-depth look, or their field guide for support in implementing a lean data approach in your organization.

Wondering how this approach looks in action? Stay tuned for my next blog on Saturday, December 7th where we will explore a couple of case studies. See you then!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

8 thoughts on “How a Lean Data Approach Can Change Your Effectiveness as an Evaluator by Elizabeth DiLuzio”

  1. Jacqueline samuels

    Elizabeth this was an interesting article that provided such useful information as it relates to data collection. The entire process of conducting an evaluation can really be quite daunting. You mentioned the issue of time as it relates to the collection of data needed to make informed decisions. Indeed the longer it takes to collect the data, the more challenging it becomes to make keen recommendations especially where time sensitivity is an issue for the collection and dissemination of the evaluation results. I like that lean data simply focuses on the targeted, relevant and quality data that is collected through efficient use of resources. It’s much like getting to the meat of the matter. Thanks for shedding some light.
    I love your Rad resource and I definitely will be checking this out. Thanks again!

    Jacqueline

    1. Elizabeth DiLuzio

      Jacqueline, I definitely agree that it is like getting to the meat of the matter. I’m glad to hear that the article was helpful and that the approach resonates with you!

  2. Hello! My name is Jessica and I am a Program Evaluation student in the Master of Education program at Queen’s. I see that one of my peers have also responded to this article, and I’m not surprised because it is really helpful for what we are currently focusing on in this course!

    First of all, thank you for posting the lean data field guide. I saved it to my computer and will be using it in the future! I actually recently applied for a position in a local college where I may need to assist in some program evaluations, so this may end up being very handy!

    I think that by utilizing the lean data approach we can really keep on track with what is necessary and ensure we are staying within budgetary limitations which are the reality in real-world evaluations. Being as impactful as possible in our evaluations is essential to providing something of value and something which will genuinely achieve an end goal.

    In my current social program evaluation, I think that I need to focus more on a lean data approach and cut away the unnecessary bloat in the data collection process.

    Thanks again for the article, and thanks for helping us students to reflect upon our practise!

  3. Hi Elizabeth,

    I am so glad that I read your article, especially today. I am currently in the process of identifying data collection methods and approaches to analysis for the program evaluation that I am designing as a part of a program evaluation course I am in at Queen’s University.

    I found this information about lean data especially helpful in my case, as I am looking at my social program in terms of an impact evaluation – is the program achieving is goals? Is the program actually having an impact on the participants and making a difference? This approach to data collection interests me a lot – particularly due to the fact that it is simple, to the point, and cost effective. The link that you posted to The Lean Data Field Guide is also very helpful – it really breaks down data collection into its simplest form and makes it easy to understand. The Question Sets that are provided are also great – I plan to use some of these and alter them to fit in with my program evaluation and survey/questionnaire questions. How much time does the lean data approach typically take to implement and carry out in comparison to a regular impact evaluation? Do you have any tips/resources in regard to the “Take Action” stage? Are their some best practices?

    I look forward to your next post in December – thanks for sharing!
    Alyssa Hall

    1. Elizabeth DiLuzio

      Hi Alyssa,

      I’m so thankful the post and its resources were helpful to you! There’s no strict time frame for a lean data assessment, but I think the trend is 4-6 weeks. Traditional impact evaluations can take a year or more. It’s quite a difference!

      As for the take action stage, there’s no prescribed way to proceed. When beginning a lean data project, it’s your desired action that drives the questions you ask and the data you collect. Once you have the data, it should point directly to your next steps. For example, if I’m trying to figure out whether or not a service I’ve been offering for a year has been reaching its intended audience or if I need to change my enrollment approach, I’ll collect data on the service’s recipients. That information, in turn, will impact whether or not I need to adapt our recruitment practices. Does that make sense?

      Best,
      Liz

Leave a Reply to Elizabeth DiLuzio Cancel Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.