My name is Alex Fink and I’m a Research Fellow in Youth Studies at the University of Minnesota. I’m a developmental evaluator with youth-serving organizations. I’ve watched the last five years as many organizations are asking youth workers to collect more and more data about program participants. As one worker I spoke to said, “We know where kids are and what they are doing from when they wake up to when they go to sleep.”
More and more, this data is analyzed by machines – algorithms that “learn” by finding patterns. Such databases are becoming an easy sell – they track clients and provide fast data analysis that “replaces” some of the work evaluators used to do. They even provide recommendations for program improvement. This is “big data” entering the realm of evaluation.
I’ve been thinking a lot about all this data. What really are we collecting? Who uses it, and what for? What role do evaluators have when all this data is processed by machines? What difference does it make – for the organization, partners, funders, and most importantly, for young people? I’ve posed the latter question to many youth I’ve collaborated with and gotten some pretty consistent answers:
Did they know all this data was collected about them? No.
Do they feel it has an impact on their lives? Yes – they’ve got a lot of people “in their business” in ways that often feel overbearing.
What do they want instead? Relationships with trustworthy and caring adults. Every young person I’ve talked to said they alone possessed the “big data” about themselves – data a machine would never know. It was data like, “the reason I’m doing poorly in school is because my teacher is racist.” Contrary to popular stereotypes, every one of these youth said they wanted more adults to know about these things. But they didn’t trust most adults to understand or care.
Lessons Learned: Evaluators have the opportunity to help organizations see the importance of developing and maintaining relationships with young people (and all clients). “Getting more data” is a big trend, but as we all have experienced, an interview will tell you a lot of things a survey cannot. Evaluators can help organizations hold true to their missions when it comes to collecting data.
Evaluators can also help show the complexity of the lives our clients often lead, raising stories that are otherwise missed in the quantitative data. This task will become more important than ever as these stories are increasingly folded into the categories that databases make available to organizational users.
Hot Tip: When working with an organizations’ clients, try asking a few questions that give clues about the data that might be useful for the organization to collect. Things like, “What could someone at this organization know about you that would help them make your life better?” and “What would someone here have to do for you to trust them enough to tell them that kind of information?”
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on theaea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
It is interesting to me and also not unexpected that the issue of personal data at a youth-serving organization is juxtaposed with concerns for “relationships with trustworthy and caring adults”. In an age where data and information about us is routinely collected and used for a wide range of purposes, one can almost lose site of the personal, human component. Indeed, I have found in my work in educational assessment and program evaluation that the issue of building trusting relationships is central. I believe that it is not only possible to have the best of both worlds (i.e. use of quality data and developing human connections), it is desirable! As the author states: “Evaluators have the opportunity to help organizations see the importance of developing and maintaining relationships with young people (and all clients).” My organization, the Association for the Cooperative Advancement of Science and Education (www.acase.org) is a nonprofit based in Saratoga Springs, NY providing consultation and professional development in assessment and evaluation to enhance education and training programs. We have explicitly identified this building of relationships as one of our learning goals for educational assessment and evaluation: that assessment data is applied to build community. For us, assessment refers strictly to the extent to which a person has attained a specific learning goal. We are interested in supporting the development of worthy capabilities in people and have a method for collecting very specific assessment information and then using it for evaluative purposes, always in the service of and interest in developing the learner. But it is also used for decision-making purposes and to add value to or improve the educational program. While the author is talking about program evaluation in general, our approach to educational program evaluation is consistent with this in many ways. By carefully and sensitively focusing on and collecting sound information, we hope to make more visible and explicit the internal process of learning that goes on in a human. Our book, Knowing the Learner (http://educationalrenewal.org/forum/knowing-the-learner/) goes into this in more depth, but the main premise is that through assessment information you can approach the educational program with intentionality and know the learner, responding to their intellectual and developmental needs. Adopting this level of precision, transparency, and fairness in the collection and use of sound data and information, the evaluator can also know the client, or, to use the words of the author: “Evaluators can also help show the complexity of the lives our clients often lead, raising stories that are otherwise missed in the quantitative data.” I thank the author for confronting the fear and trepidation that evaluation in general, and collection of personal data in particular, rouses in us. I hope that we can acknowledge the power and opportunity in using (assessment) data and information to build community and ultimately bring value to the stakeholders of these programs and organizations.
Great summary of the limitations of big data and the importance of asking thoughtful questions. Thanks!
But didn’t we go to Big data, because individual interviews were getting too time-consuming, expensive, cumbersome, unreliable and uncomfortable. We did not and probably still do not have the luxury of time to establish trust and then reach into the deepest crevices of human minds. That is ideal yes, but is it applicable on a large scale?