Ed Eval TIG Week: The Parallels Between Evaluation Methods and Blank’s Lean Startup Methods by Cathy Callow-Heusser

Hello! I’m Cathy Callow-Heusser. Although I started my career as a software engineer, I’ve been an evaluator for nearly 30 years, working in academia, industry, and as an independent evaluator. I am also a math educator and early literacy specialist.

Two years ago, I co-founded a company to develop software solutions for education. In January, we were awarded a National Science Foundation (NSF) Small Business Innovation Research (SBIR) Phase 1 grant to develop knowledge-based software tools – aka an “expert system” – to guide story writers in developing early literacy curricula in any language. Our key goal is to provide support to develop early literacy curricula for endangered and underserved languages, or for people who want to develop culturally-bound curricula in English.

Once we were awarded the SBIR grant, NSF required us to attend their lean startup methods bootcamp based on Steve Blank’s methods for building successful products and companies. I felt like I was learning about conducting needs assessments or appreciative inquiry under a different name. Blank’s methods and checklists felt comfortable and familiar for someone involved in evaluation as long as I have been. Yet, these methods aren’t applied to a product that is already developed and being evaluated using formative evaluation methods to improve the product, or summative  evaluation methods to determine its effectiveness or impact. Rather, these evaluative methods are used prior to product development to ensure the product meets customer needs, to test hypotheses regarding product design and potential customer segments, and to ensure a scalable, viable, salable product.

As part of the bootcamp, NSF required us to conduct a substantial number of “customer discovery” interviews. We were amazed at what we learned from potential customers before we ever started product development, and the interviews resulted in substantial “pivots” to our design—long before we began coding.

As we’ve gone through this evaluative hypothesis-testing process, I have continually reflected on its parallels to the evaluation methods I’ve used for decades. Imagine how much better our curricula for literacy, math, science, social studies, trauma-informed care, character education, sex education, or any other content area would be if we engaged potential customers in this process BEFORE development was started?

Hot Tip: I think Steve Blank is onto something we need as educators and evaluators – something that feels like a comfortable old shoe to this evaluator. I’m astounded at how much better our final product is likely to be because we engaged in using these lean startup methods long before we began development. I’ll definitely add Blank’s lean startup methods to my evaluation toolkit!

Rad Resources:

Steve Blank’s Lean Startup website has invaluable free resources for learning about these methods.

The Startup Owner’s Manual by Steve Blank and Bob Dorf (2012) includes many checklists to guide the customer discovery process.

Why the Lean Startup Changes Everything by Steve Blank (2013) gives a brief overview of lean startup methods.

What’s Next for Steve Blank and the Lean Startup by Greg Satell describes Steve Blank’s development of lean startup methods.

The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PreK-12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.