Welcome to aea365! Please take a moment to review our new community guidelines. Learn More.

Research on Evaluation (RoE) TIG Week: “Whether They Say It or Not, a Lot of the Program Evaluation is Informal”: Perspectives and Reflections from a Pilot Study of a Grassroots Organization by Christine Liboon

Hello, AEA365 readers!

Christine Liboon

My name is Christine Liboon and I’m a doctoral candidate in the Social Research Methodology program at the UCLA School of Education and Information Studies. I am currently the Program Co-Chair for the Graduate Student and New Evaluators TIG (GSNE) as well as Member-at-Large for the Research on Evaluation TIG (RoE). I’m excited to share a snippet of my first RoE pilot study during Year 1 in graduate school. In this short blog, I’ll offer:

  • what an RoE study can look like in the context of a small organization;
  • tips on the research process for evaluation researchers;
  • resources for practitioners in a grassroots context.
Context of RoE Study and Organization Perspectives
Executive Director quote

The Executive Director’s quote supports how staff defined evaluation in a women’s English as a Second Language (ESL) vocational educational permit training program at a grassroots refugee resettlement organization. My motivation for this study was to understand non-formal educational programs and how program improvement occurs in these smaller (but mighty!) spaces.

I was also interested in understanding evaluation from a service-delivery perspective (i.e., practice) and what the organization’s leadership and staff defined to be successful (i.e., impact) in a context that often initially works with limited infrastructure or resources to conduct a full-scale evaluation.

In my qualitative case study, I wondered: how do program staff define and use evaluation results to provide continued support for young and adult refugees? The Community Outreach Coordinator reflected on the informality of evaluation in their work: 

Executive Director quote

Both vignettes above highlight how the organization sees evaluation through informal moments: conversations with clients, frequent discussions with each other and the board, and planning with feedback gathered to make improvements in their program. As a researcher, I gathered information and built trust through interviews with the Executive Director, the Community Outreach Coordinator, the Program Driving Instructor, and the Communications Manager. Conversations which each individual helped me to deeply understand their perspectives on these informal moments and how they re-defined evaluation (for me). 

Lessons Learned

Critical Reflexivity. In reflection, questions emerged on the research process and my research practice that made me think: whose voices did I privilege and did I miss? Was there a critical way of looking at how staff defined “informal”? Could I have included program participants, such as the students, in my data collection to more equitably define what informal evaluation meant in this space? Did I center race/ethnicity, historically marginalized voices, and power dynamics enough? Did I possibly miss something or forget to ask anything important?

The Beauty of Iteration. New to the field in conducting an RoE study, I was eager to accept the first thing I discovered as the answer. I learned patience to allow for emergence and with iteration, we can arrive at a richer conclusion. Research is a process. It requires framing, re-framing, and a re-connection to how questions shape what we learn in grassroots organizations and what we define as an evaluation practitioner. It is inherently tied to reflexivity (which can be a luxury for practitioners). Questions that emerge as a researcher require constant reflection to continuously improve how research and practice is done.

Rad Resources

Here are a couple of resources that might be useful for grassroots evaluation practitioners:

Do you work at a grassroots or small-scale non-formal educational organization? If so, how do you and your colleagues define evaluation, if at all? Are you interested in doing RoE? If you’d like to continue the conversation, reach out! LinkedIn

The American Evaluation Association is hosting Research on Evaluation (RoE) Topical Interest Group Week. The contributions all this week to AEA365 come from our ROE TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.