Hi, this is Ally Rakus. I’m a graduate student of public policy at NYU Wagner and intern at Evaluation + Learning Consulting. The COVID-19 pandemic has created new and exacerbated existing challenges of administering programs and services; program evaluation is not immune to these challenges, especially data collection. Under normal circumstances, collecting data from participants is relatively easy when participants are regularly on site. However, with many in-person gatherings postponed or moved online, the shift to remote data collection emphasizes unequal access to technology by participants.
In response, we must ensure equity in our data collection methods. A few ways to ensure representation of all participants, regardless of technology access, are knowing your population, trying new methods, and triangulation.
Knowing Your Population
Knowing about the people from whom you want to collect feedback is always important, but is especially important now. We are evaluating programs and collecting participant data under special circumstances, including a global pandemic and racism protests. There are many questions to ask ourselves and our participants, including:
- What does access to technology look like for our population?
- How can we diversify our approach to make sure we are collecting from all sub-groups?
- How are our participants affected by the current crises (including unemployment/job instability, childcare, physical and mental health)?
UN Women and the World Health Organization provide recommendations for collecting data about violence against women and girls; however, these recommendations can be applied to similar populations.
- Do not proceed with data collection if there are any risks of harm to the respondents
- Choose the most appropriate data collection method and source for your context and objectives, always ensuring the safety of respondents
- Advocate for the needs of marginalized groups within your population
Trying New Methods
To account for people without internet or technology access, we may need to try data collection methods that aren’t necessarily our go-to’s, such as text and voice surveys. Utilizing automated text messaging (SMS) and calling (IVR – interactive voice response) to administer surveys is an option for those without internet access.
RTI International recommends SMS and IVR surveys during the pandemic; the organization focuses on the population of low-middle income countries. They provide the following tips for these surveys:
- Choose your mode wisely – consider literacy of your population and cost per method
- Send reminders … lots of them
- Incentivize your respondents
- SMS and IVR work best for surveying list samples
- Check your sample – confirm validity of contact information
- Mix modes to improve quality
- Run a pilot first
Qualitative data, such as focus groups and interviews, and secondary data sources may be logistically convenient in some instances for data collection. The focus groups and interviews could be conducted via a phone call or an online video conference, depending on technology accessibility. Secondary data may be a viable option if primary data will be inaccurate or is difficult to obtain given current circumstances.
No matter what method you choose, methodological triangulation is an important evaluation approach to consider. Because it validates your data by comparing results across two or more sources, it helps to protect against common research biases that arise when using only one sampling method, such as measurement and sampling biases.
These recommendations are meant to serve as a guide for evaluators to consider when prioritizing equity and proper representation in data collection during the pandemic.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.