Hello, AEA365 readers!
My name is Christine Liboon and I’m a doctoral candidate in the Social Research Methodology program at the UCLA School of Education and Information Studies. I am currently the Program Co-Chair for the Graduate Student and New Evaluators TIG (GSNE) as well as Member-at-Large for the Research on Evaluation TIG (RoE). I’m excited to share a snippet of my first RoE pilot study during Year 1 in graduate school. In this short blog, I’ll offer:
- what an RoE study can look like in the context of a small organization;
- tips on the research process for evaluation researchers;
- resources for practitioners in a grassroots context.
Context of RoE Study and Organization Perspectives
The Executive Director’s quote supports how staff defined evaluation in a women’s English as a Second Language (ESL) vocational educational permit training program at a grassroots refugee resettlement organization. My motivation for this study was to understand non-formal educational programs and how program improvement occurs in these smaller (but mighty!) spaces.
I was also interested in understanding evaluation from a service-delivery perspective (i.e., practice) and what the organization’s leadership and staff defined to be successful (i.e., impact) in a context that often initially works with limited infrastructure or resources to conduct a full-scale evaluation.
In my qualitative case study, I wondered: how do program staff define and use evaluation results to provide continued support for young and adult refugees? The Community Outreach Coordinator reflected on the informality of evaluation in their work:
Both vignettes above highlight how the organization sees evaluation through informal moments: conversations with clients, frequent discussions with each other and the board, and planning with feedback gathered to make improvements in their program. As a researcher, I gathered information and built trust through interviews with the Executive Director, the Community Outreach Coordinator, the Program Driving Instructor, and the Communications Manager. Conversations which each individual helped me to deeply understand their perspectives on these informal moments and how they re-defined evaluation (for me).
Lessons Learned
Critical Reflexivity. In reflection, questions emerged on the research process and my research practice that made me think: whose voices did I privilege and did I miss? Was there a critical way of looking at how staff defined “informal”? Could I have included program participants, such as the students, in my data collection to more equitably define what informal evaluation meant in this space? Did I center race/ethnicity, historically marginalized voices, and power dynamics enough? Did I possibly miss something or forget to ask anything important?
The Beauty of Iteration. New to the field in conducting an RoE study, I was eager to accept the first thing I discovered as the answer. I learned patience to allow for emergence and with iteration, we can arrive at a richer conclusion. Research is a process. It requires framing, re-framing, and a re-connection to how questions shape what we learn in grassroots organizations and what we define as an evaluation practitioner. It is inherently tied to reflexivity (which can be a luxury for practitioners). Questions that emerge as a researcher require constant reflection to continuously improve how research and practice is done.
Rad Resources
Here are a couple of resources that might be useful for grassroots evaluation practitioners:
- SLP4i: Philosophies and Approaches
- Level Best: How Small and Grassroots Nonprofits Can Tackle Evaluation and Talk Results (see review here)
Do you work at a grassroots or small-scale non-formal educational organization? If so, how do you and your colleagues define evaluation, if at all? Are you interested in doing RoE? If you’d like to continue the conversation, reach out! LinkedIn
The American Evaluation Association is hosting Research on Evaluation (RoE) Topical Interest Group Week. The contributions all this week to AEA365 come from our ROE TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.