Hi, I’m Betsy Block, Principal & Founder of B3 Consults, and I’m excited to kick of this week talking about ensuring you’ve got GREAT DATA. I define “great data” as accurate, timely and most of all useful. As evaluators we want to focus a lot on using data, but we often have to help mission-driven organizations do a better job of collecting it in the first place, and solving the technical mysteries in how to do it. The “best” data system alone won’t magically procure great data. I hope this week’s blogs give you insight and tools to help you collect great data.
In my time as a “pre-valuator” and coach, I have watched mission-driven organizations struggle to evaluate their work – or even want to pursue evaluation – because their data systems just fail them. These failures are usually preventable, if only these organizations had a better resource on setting up their data to start.
Representatives of amazing mission-driven organizations, software vendors and foundations spoke to my colleagues and me over the last couple of years about their success (and failures) in data system implementations. And regardless of their organizational perspective, these folks all seem to land on similar advice in making sure the data system actually helps the organization capture great data.
Hot Tips:
- You need a map to get where you are going. Those logic models and theories of change? They can help you to make sure that you have clear directions and priorities for the data you collect.
- Get out of the silos. We all want to load up one person with magically making it happen – but as evaluators, we are skilled at engaging multiple perspectives, and this is a great time to use that skill.
- You can get it fast, you can get high quality, or you can get it cheap… but not all three. Take your time, focus on quality, and you can control costs.
- Be patient and compassionate. Some of the changes will push against longstanding ways of doing business, and people will have unanticipated emotions about change.
- Start with WHY. It seems obvious, but many organizations pursue getting a new data system without articulating what is going to get better when they do. Think of it as beginning with the ending: know what success will look like!
Rad Resources:
- Making Wise Decisions offers user-friendly, step-by-guidance to organizations that are considering adopting a new data management system – including free-to-use templates. No need to recreate the wheel!
- Salesforce Getting Ready Playbook by Sam Dorman Consulting. Though Salesforce focused, this playbook includes lots of thoughts applicable to all organizations around what to consider before selecting a system.
- K. Kellogg Foundation Logic Model Development Guide by W.K. Kellogg Foundation. Practical and clear guidance on how to create a logic model, always a great go-to resource.
The American Evaluation Association is celebrating Getting Great Data Week. All posts this week are contributed by evaluators who came together to write about the theme of getting data that is accurate, timely and most of all useful. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hi Betsy,
I really enjoyed reading your article. I have found it very informative and relevant in my current role as an elementary public school teacher. I have participated in many program evaluations over the past several years, and the ‘hot tips’ that you have provided can serve as a framework that leaders can use in order to evaluate the effectiveness of a program.
I agree that the team must have an understanding of what is required in order to have a clear focus. The team needs to critically think of data collection that needs to occur and how this can be used in program evaluations. I have had the privilege of working with teams that knew what data sources needed to be collected to evaluate the program. This was done during the initial planning stage. On the other hand, I have worked with groups that lacked direction from the leader and hence were unsuccessful in gathering effective sources of data due to the lack of a clear plan. You stated the importance of collective team work and the inclusion of multiple perspectives within a program evaluation. This is important as everyone in the group needs to share the work load and there needs to be a clear line of communication established by the leader. When facilitating a program inquiry, I have used the Seven Norms of Collaboration to help guide the evaluative process.
These norms include:
Pausing: Thought processing time prior to responding to a question
Paraphrasing: Sharing ideas, restating, going over the facts
Posing Questions: Asking relevant questions
Putting Ideas on the Table: Collaboration, building consensus
Providing Data: Qualitative and Quantitive data sources to help the group moving forward with the plan or for evaluation purposes
Paying Attention to Self and Others: Listening to everyone and understanding the different learning styles of the group
Presuming Postive Intentions: Assuming that everyone has positive well intentioned ideas moving forward
You also stated that quality data is essential, I agree, that in the educational context, educators need to collect data that reflects student growth over the time of the program and to assess if overall growth has been made. This will determine possible next steps for program delivery. When educators select pertinent data from varied sources, it makes the program evaluation much stronger as opposed to data that fails to provide a comprehensive picture.
The Foundation Logic Model Development Guide resource that you have provided is a very comprehensive guide that provide pertinent information when conducting a program inquiry. I have perused the resource, and there are valuable tips that can be shared with the team prior to conducting a program evaluation to provide a clear pathway. There is an exercise checklist in Chapter (Page 20), this checklist provides criteria for the group when reflecting on the outcomes and impact of the program. This exercise provides a logical framework that the group can work with and can be further tweaked to meet the groups targets.
In closing, as a future school administrator, I am looking forward to working with my team members by helping them develop a comprehensive approach to initiating a program and to conduct an effective program evaluation by critically thinking of sources of data and how they will help determining the overall outcome of a program.
Ian Maclure
Hi Betsy,
Thanks a lot for the information on getting GREAT DATA. The step by step tip is quite clear and simple to understand. As a researcher this information you shared would help me navigate through my research.
Thanks
Ade