Howdy, fellow evaluators! We are Morgan Buras-Finlay and Eric Barela, Salesforce.Org’s Measurement & Evaluation team. In our post today, we will share what we’ve learned through our experience working in and providing support to the nonprofit sector, especially direct service and education organizations.
In our experience, organizational data quality is most often a reflection of work process – what staff are doing and how they are doing it. We are going to share some of the challenges we’ve seen with organizational data quality and tactics for assessing it. There are a lot of checklists out there, but we have found that they don’t always suit our needs, and so we have to think on our feet about good ways to assess data quality. Essentially, we want to share how we live AEA Evaluator Competency 4.3: The competent evaluator manages and safeguards evaluation data.
The day-to-day data that an organization collects through the running of programs are often collected through forms and by staff, sometimes on paper, as a result of program participation. Common data quality issues include completeness of the data, recurring gaps in the data, accuracy and utility.
One solution we have found to uncover issues is to ask staff to walk us through each step they take to collect, enter and record data. This helps to uncover the major pain points and inconsistencies that can lead to issues of data quality. It’s important to do this consistently, with both new and veteran staff members. While most staff will regularly monitor their own data quality, everyone has a bad day. Consider yourself to be the data quality safety net.
For data that is collected through surveys and assessments, the issues of completeness, accuracy and utility certainly apply. However, it’s also important to look for whether certain voices are being heard louder than others and why that may be. We have found that many issues with survey data quality emerge from the methods themselves. An example is a large municipal survey that was intended to gather information from all residents in a city. The 15-30 minute survey was only administered online, and there were delays in launching the non-English survey sites. As a result, non-English speakers and those without computer access were dramatically underrepresented, and the results, which were intended to inform city-wide service delivery, were challenged on their accuracy.
- Narrow the focus of what is being collected, and add data points as quality improves.
- Collect information at the right time, in the right way, in the right language for participants or staff.
- Host a data party! Facilitate review of questions/instrument by community and staff to make sure the right questions are being asked in the right way.
- Invite community members and staff to view the results/data, and share their perspective on whether it is jiving with their experiences
Rad Resource: AEA Evaluator Competencies
The American Evaluation Association is celebrating Getting Great Data Week. All posts this week are contributed by evaluators who came together to write about the theme of getting data that is accurate, timely and most of all useful. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.