Ed Eval TIG Week: Sheila A. Arens & Mariana Enriquez on Alternative Data Collection Methods: One Size Does Not Fit All

Hello! We are Sheila A. Arens and Mariana Enriquez.

Evaluators frequently propose psychometrically sound instruments or low-inference observation protocols used by trained observers for data collection. Such instruments and methods are sometimes the most appropriate to address a question of interest. However, we believe it is important for evaluators to stretch beyond traditional data collection methods when it makes sense to do so and when traditional, quantitative approaches will not be able to reach the target population, capture their experiences, or produce meaningful, reliable data.  It is important to give careful consideration to how and whether respondents will engage in data collection, and even when traditional methods may be desirable, to think about whether there are more appropriate ways of delivering the data collection request to the target population. In other words, data collection cannot be a one-size-fits-all approach.

We frequently work with populations for whom completing surveys or participating in focus groups would be difficult. In fact, these data collection methods may even further disenfranchise these individuals. For instance, some may feel immense discomfort discussing any issues in a group, or may quickly identify whose lead they need to follow in the group of participants to remain in a “safe place” … yet, the evaluator may persist in using focus groups as the means of data collection.

Lesson Learned: Recognizing that data collection cannot be a one-size-fits-all approach, we have sought alternative ways to engage individuals. We must be nimble and creative in our data collection approaches. And, although alternative methods may not work for everyone, how the target population is informed and engaged may make a difference in their participation. Including “participants” in the process to decide the best ways to reach them—and being humble about what we know, what works, what’s best, etc.—not only seems prudent but also seems like a culturally competent approach.

Alternative data collection approaches might include approaches that are similar to focus groups but provide additional opportunities for sharing not circumscribed by a single facilitator or necessarily hampered by group dynamics, the use of photographs or images produced and annotated by participants (photovoice, for instance: http://steps-centre.org/methods/pathways-methods/vignettes/photovoice/ ), or online social networks such a Twitter or Google Hangout.

A word of caution: Be creative, but be sure that your data collection method is appropriate for the evaluation questions of interest, is sensitive to participants’ needs and existing resources, and that the evaluation budget can support the additional burden of a potentially more time-consuming analytic method.

Rad Resources:

  • PhotoVoice, an app for iPhones (respondents can upload images and record their thoughts about the image)

The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PK12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.