Hello! We are Sheila A. Arens and Mariana Enriquez.
Evaluators frequently propose psychometrically sound instruments or low-inference observation protocols used by trained observers for data collection. Such instruments and methods are sometimes the most appropriate to address a question of interest. However, we believe it is important for evaluators to stretch beyond traditional data collection methods when it makes sense to do so and when traditional, quantitative approaches will not be able to reach the target population, capture their experiences, or produce meaningful, reliable data. It is important to give careful consideration to how and whether respondents will engage in data collection, and even when traditional methods may be desirable, to think about whether there are more appropriate ways of delivering the data collection request to the target population. In other words, data collection cannot be a one-size-fits-all approach.
We frequently work with populations for whom completing surveys or participating in focus groups would be difficult. In fact, these data collection methods may even further disenfranchise these individuals. For instance, some may feel immense discomfort discussing any issues in a group, or may quickly identify whose lead they need to follow in the group of participants to remain in a “safe place” … yet, the evaluator may persist in using focus groups as the means of data collection.
Lesson Learned: Recognizing that data collection cannot be a one-size-fits-all approach, we have sought alternative ways to engage individuals. We must be nimble and creative in our data collection approaches. And, although alternative methods may not work for everyone, how the target population is informed and engaged may make a difference in their participation. Including “participants” in the process to decide the best ways to reach them—and being humble about what we know, what works, what’s best, etc.—not only seems prudent but also seems like a culturally competent approach.
Alternative data collection approaches might include approaches that are similar to focus groups but provide additional opportunities for sharing not circumscribed by a single facilitator or necessarily hampered by group dynamics, the use of photographs or images produced and annotated by participants (photovoice, for instance: http://steps-centre.org/methods/pathways-methods/vignettes/photovoice/ ), or online social networks such a Twitter or Google Hangout.
A word of caution: Be creative, but be sure that your data collection method is appropriate for the evaluation questions of interest, is sensitive to participants’ needs and existing resources, and that the evaluation budget can support the additional burden of a potentially more time-consuming analytic method.
Rad Resources:
- PhotoVoice, an app for iPhones (respondents can upload images and record their thoughts about the image)
- WorldCafe design principles
The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PK12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hi Sheila and Mariana,
I appreciate your post as it brings attention not only to alternative methods for data collection, but reminds us of the importance of cultural sensitivity that should be taken into account when designing methods for data collection. I am a new evaluator and was drawn to your post in order for me to gather more information about data collection and how data collection can strengthen my program evaluation. What I particularly found valuable in your post was that you identified that evaluators must reflect upon the data collection process. In order for evaluators to make decisions about the appropriateness of “traditional methods” and if they are appropriate, evaluators must still recognize how participants are engaging with the collection methods and how this might impact the validity of the data. I really agree with your statement, “data collection cannot be a one-size-fits-all approach.” This is such a powerful statement to remember when developing an evaluation process. As you suggest, taking into account the type of population, i.e. background, demographics and social economic status, that is involved in providing the data should recognized. Reflecting on the data we collect from various populations, demographics and cultural backgrounds is important as the data process can be impacted depending on the method one chooses.
As you suggest, being sensitive to the participants and how they will interact with the data collection method would be important to know as it can have an impact on collecting valuable and reliable data. I also appreciate your comment:
Be creative, but be sure that your data collection method is appropriate for the evaluation questions of interest is sensitive to participants’ needs and existing resources and that the evaluation budget can support the additional burden.
I feel that there must be a fine balance between choosing an appropriate data collection method while still taking into account other aspects such as the demographics or cultural backgrounds of the participants.
Your post was very helpful in giving me a starting place and some pieces to reflect on while creating my own program evaluation and especially when choosing methods to collect data that is valuable, reliable and provides information that can possible lead to positive change in the program.
– Michelle