Howdy, fellow evaluators! We are Morgan Buras-Finlay and Eric Barela, Salesforce.Org’s Measurement & Evaluation team. In our post today, we will share what we’ve learned through our experience working in and providing support to the nonprofit sector, especially direct service and education organizations.
In our experience, organizational data quality is most often a reflection of work process – what staff are doing and how they are doing it. We are going to share some of the challenges we’ve seen with organizational data quality and tactics for assessing it. There are a lot of checklists out there, but we have found that they don’t always suit our needs, and so we have to think on our feet about good ways to assess data quality. Essentially, we want to share how we live AEA Evaluator Competency 4.3: The competent evaluator manages and safeguards evaluation data.
The day-to-day data that an organization collects through the running of programs are often collected through forms and by staff, sometimes on paper, as a result of program participation. Common data quality issues include completeness of the data, recurring gaps in the data, accuracy and utility.
One solution we have found to uncover issues is to ask staff to walk us through each step they take to collect, enter and record data. This helps to uncover the major pain points and inconsistencies that can lead to issues of data quality. It’s important to do this consistently, with both new and veteran staff members. While most staff will regularly monitor their own data quality, everyone has a bad day. Consider yourself to be the data quality safety net.
For data that is collected through surveys and assessments, the issues of completeness, accuracy and utility certainly apply. However, it’s also important to look for whether certain voices are being heard louder than others and why that may be. We have found that many issues with survey data quality emerge from the methods themselves. An example is a large municipal survey that was intended to gather information from all residents in a city. The 15-30 minute survey was only administered online, and there were delays in launching the non-English survey sites. As a result, non-English speakers and those without computer access were dramatically underrepresented, and the results, which were intended to inform city-wide service delivery, were challenged on their accuracy.
Hot Tips:
- Narrow the focus of what is being collected, and add data points as quality improves.
- Collect information at the right time, in the right way, in the right language for participants or staff.
Cool Tricks:
- Host a data party! Facilitate review of questions/instrument by community and staff to make sure the right questions are being asked in the right way.
- Invite community members and staff to view the results/data, and share their perspective on whether it is jiving with their experiences
Rad Resource: AEA Evaluator Competencies
The American Evaluation Association is celebrating Getting Great Data Week. All posts this week are contributed by evaluators who came together to write about the theme of getting data that is accurate, timely and most of all useful. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
You have pointed out the issue of completeness and accuracy as well as a utility when the data is collected through surveys.
Dear Morgan and Eric,
Thanks very much for posting this information in regards to quality data collection. I am currently taking a program evaluation course at Queens University and planning to launch a program evaluation for an international student orientation program. As you mentioned in the post, the data quality is the reflection of the work process. The collected data tell you what, why and how the staff is doing. When evaluator(s) are assessing the data, there are also challenges when doing the collection. Such as the staff members refuse to participate and unwilling to share the real feeling of the program. The fear of isolation during the data collection process will lead the participants to feel uncomfortable. Especially when the evaluation questions or interviews are related to some sensitive information. As you mentioned about the organization data collection, the data is collected through forms on paper by staff. You have also indicated the solution for solving data collection issues, and to make sure the information has been collected consistently would be to ask the staff to walk with you through each step.
You have pointed out the issue of completeness and accuracy as well as a utility when the data is collected through surveys. The problem sometimes occurs because of the methods. Thus, when people are doing an online survey, I believe the survey format shall be easy to understand and concise. The type of questions asked has to be carefully considered before launching the survey. Also, we have to create a relatively specific survey that asks questions related to a critical topic instead of running around just like you said: “Narrow the focus of what is being collected.”
Could you please elaborate a little bit more about collecting information at the right time, in the right way? From my understanding, choose the right time is very important, do we need to provide the flexibility for the participants to finish the survey but with an identified deadline? The reluctance of doing the survey will arise if we give the participants too many flexibility.
I appreciate that you have mentioned hosting a data party to review questions and to make sure the right questions. From the Better Evaluation, data party is commonly used for participatory data analysis. The process usually including a “gallery walk” with informally information discussion and a “world café” where people are focusing on the particular topics. My question here is who we should include into these data parties? At what stage of data collection that the data party can be happening? Does data party suit for all kinds of programs or only feasible for some specific programs? Are there any drawbacks for data party?
Again, thanks very much for sharing your experiences and your tips when doing the data collection. I found your suggestions are straightforward to follow and helpful. I am looking forward to your replies. Thank you.
Jing
References:
Data party. (n.d.). Retrieved from https://www.betterevaluation.org/en/evaluation-options/data_party
this is a great pieces of information