Hello, AEA365 community! Liz DiLuzio here, Lead Curator of the blog. This week is Individuals Week, which means we take a break from our themed weeks and spotlight the Hot Tips, Cool Tricks, Rad Resources and Lessons Learned from any evaluator interested in sharing. Would you like to contribute to future individuals weeks? Email me at AEA365@eval.org with an idea or a draft and we will make it happen.
Hello. From 2016 to 2023, Sharon Twitty, Natalie Lenhart, and I (Paul St Roseman) have implemented an evaluation design that incorporates empowerment evaluation, development evaluation, and evaluation capacity building. This blog series, which we call Walking Our Talk, presents the process in four parts:
- Laying the Foundation – The Rise of The Logic Model
- Using the Data – Emerging a Data Informed Evaluation Design through Peer Editing
- New Insights – Emerging A Collaborative Diagnostic Tool and Data Visualization
- Charting the Course of Data Use – Collaborative Data Analysis
Today’s post is Part 4 in the series.
Piloting the ACDT
In 2019, Sharon, Natalie, and I (The ARCHES Evaluation Team) held a series of meetings that focused on developing and implementing the ARCHES Collaborative Diagnostic Tool (ACDT). It was decided that this tool would be used in a multi-year pilot with the Tulare-Kings College and Career Collaborative (TKCCC). The focus of this effort was to implement the ACDT with the goals of testing and refining the language, validating and calibrating outcome indicators, clarifying roles for collaboration, developing data visualizations, and developing protocols that support the collaborative analysis of data.
In this pilot, the ACDT was implemented three times with the leadership network of TKCCC. Throughout this process, ARCHES and TKCCC engaged in a joint effort to refine the ACDT and its accompanying codebook. We also developed the CART Analysis Cycle (Compile, Analyze, Review, Transform) to frame the process of transitioning data from collection to analysis to informed decision making. Additional documents supporting collaborative data analysis efforts included a digital workbook designed to align data with TKCCC’s strategic goals, an interactive scorecard for data exploration and insight generation, and an analytic memo summarizing trends from the data collected across the ACDT administrations. To translate data-driven decisions into practical steps, the Evaluation Team created a coaching and support plan, which resulted in the development of an ACDT Coaching and Support Workbook. The process came to be known as the ACDT Suite.
Lesson Learned: Collaborative Data Analysis Sets a Foundation for Data-Informed Decision Making
During the pilot phase of the ARCHES Collaborative Diagnostic Tool (ACDT), Collaborative Data Analysis emerged as a process that melded three areas of skill development: (1) achieving clarity and consensus in contextualizing data, (2) establishing group norms for reviewing and discussing collected data, and (3) formulating explicit strategies to turn data-informed decisions into concrete actions. The effort to contextualize data focused on refining the terminology within the diagnostic tool and aligning the measurement outcomes with the strategic objectives of the TKCCC collaborative. When data from the ACDT was gathered, norms were set up that outlined the collaborative approach between ARCHES and TKCCC for data review and analysis. This included specifying roles for the analysis and jointly creating meeting agendas to structure the review process. Lastly, strategies to translate data-informed decisions into action was facilitated by ARCHES’ coaching (provided by Sharon) which resulted in priorities that were aligned to TKCCC strategic plan.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.