Hello fellow evaluators! We are Amanda Sutter, Allison Prieur, Valerie Marshall, Rachael Kenney, Kari Ross Nelson, and Christine Liboon, 6 evaluation practitioners with 10 – 19 years of evaluation experience who are studying evaluation across 5 U.S. universities. We teamed up to examine our experiences learning Research on Evaluation (RoE) in our doctoral programs.
Our first decision as a team was whether to write an editorial or embark on an empirical study. Given the limited literature on doctoral students’ experience learning RoE, we opted for a study. We used a design called collaborative autoethnography (CAE) to explore our experiences, and wrote this post to spotlight CAE as an accessible and flexible option for evaluators to conduct RoE. CAE is a systematic and iterative method of inquiry that combines ethnography, collaboration, and autobiography, whereby researchers are also subjects. CAE offers a way for evaluators to investigate and document their lived experiences in any evaluation context.
CAE helped us uncover how our practitioner backgrounds, faculty support, and evaluation connections were among the biggest positive contributors to our scholarly development, and how limitations in formal RoE coursework, applied opportunities, and funding created barriers. We noted a particular tension for programs between training students to become evaluation practitioners and scholars, aligned with prior research. Our positionality as students who were already practitioners created a unique lens to see what we believe are unnecessary siloes between evaluation practice and research on evaluation. We offer many opportunities for change across the education ecosystem, including students developing reflective practice skills, programs expanding cross-institutional collaborations, and the field strengthening the practice-research relationship more broadly. We hope CAE can be used more among evaluators toward that goal. Keep an eye out for our upcoming article about this work in New Directions for Evaluation!
Lessons Learned
- CAE strengths center on facilitating collaborative processes where diverse perspectives are valued, workload is shared, and teams co-create research processes.
- CAE challenges necessitate allowing for adequate time for collective decision-making, attention to team-based processes, and the inevitable learning curve of any new method.
- Deciding to use CAE is only the beginning. There are many decisions about how to collect data (e.g., audio, video, journals, documents), what data to include (e.g., transcripts, chats, videos), and how to analyze (e.g., software, types of coding, other approaches). Make sure to follow good research practices on documenting all decisions.
Hot Tips
- CAE can include varying levels of engagement; while our approach was to have all team members involved in all research stages, other studies could have facilitators involved in different parts to offer an outside perspective.
- CAE can prioritize breadth or depth; where we focused on diverse experiences to understand variation, the study could also focus on diving deeply into shared experiences to understand similarities (e.g., participants from the same program type or background).
- CAE is especially helpful in topics that may be complex and impacted by various factors, changing and evolving over time.
Rad Resources
- Individual and Collaborative Autoethnography as Method – Book chapter (available full-text) on using CAE in a handbook on all autoethnography methods.
- Collaborative Autoethnography – Book dedicated to CAE methods, available to preview or purchase.
- Collaborative Autoethnography as Multivocal, Relational, and Democratic Research: Opportunities, Challenges, and Aspirations – Journal article on navigating CAE challenges and opportunities.
- Academic Trajectories and Success: A Collaborative Autoethnography of a Mentorship Program – Open source article on using CAE in an evaluation of a higher education mentoring program.
- Using Cognitive Complexity to Understand Role-Play as a Pedagogical Tool in Graduate Evaluation Education – American Journal of Evaluation article using CAE in evaluation education.
- Reciprocity in Research and Evaluation: Conceptualizing Utang Na Loob , Pakikipagkapwa , and Alalay as Filipina American Educational Researchers – Book chapter on using CAE to understand reciprocity in evaluation, with particular attention to cultural context.
The American Evaluation Association is hosting Research on Evaluation (ROE) Topical Interest Group Week. The contributions all this week to AEA365 come from our ROE TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.