Hello from Erin Clements, Sarah Kohari, Woody Stanley, and Jill Tirnauer! While new to Washington Evaluators, the AEA Affiliate sponsoring this week’s blog material, our team has more than 60 years of combined experience conducting performance management and program evaluation studies in the public and nonprofit sectors. Previously employed in federal government agencies and international consulting organizations, Jill, Sarah, and Woody currently work as independent consultants. Erin, who is new to the field of program evaluation, is currently employed in a nonprofit organization.
Earlier this year, the Arts Education Partnership (AEP) requested assistance from the Washington Evaluator Evaluation Without Borders (EWB) community engagement initiative to identify potential performance indicators for its programs and recommend an approach for reporting progress in meeting its mission. AEP is a national network of over 100 organizations dedicated to advancing arts education and serves as the Nation’s hub for individuals and organizations committed to making high-quality arts education accessible to all U.S. students. AEP works to accomplish its mission by growing and supporting a network of national, regional, state and local partners, who in turn engage with and influence state leaders who determine education policy and curricula.
Our team of volunteer evaluators worked with the AEP Executive Director, to respond to their needs. We were new to the EWB process and came to the task with different backgrounds and levels of experience in evaluation, which provided us an opportunity to learn from one another while also responding to AEP’s needs. Likewise, the Executive Director was new to AEP and was still taking stock of the organization as well as its awareness and support for program design and performance management practices.
Through a series of virtual conversations with the Executive Director, our team garnered her feedback at each stage of the consultation and learned more about how we could best support and improve monitoring, evaluation, and reporting practices in AEP. We learned that the task of evaluating AEP program performance is made more difficult due to its networking and convening role whereby it works indirectly, through its partner organizations, to achieve its desired long-term outcomes.
During our meetings we served as a sounding board and coach for the Executive Director as AEP initiated its first steps towards program design and performance management – a survey following a webinar with partners to better understand their wants and needs. Furthermore, we provided AEP with a framework for its programs and their ongoing evaluation that can be shared with staff, partners, and others during the coming months. Although this was just the first few steps, as a result of EWB, AEP is on their way to ingraining an evaluation mindset into the organization. We were glad that we had a chance to help them get off to a good start!
Hot Tip:
Guided by the need to assess tangible results as well as long-term outcomes, we advised AEP on an approach that first solidified the overall program logic and from there derived key output and outcome indicators that could be used to assess the value and utility of the network as well as the network’s influence over state education policies. We provided several deliverables, including a program logic model, a sphere of influence analysis, and a list of possible program output and outcome indicators that included indicator definition/descriptions, data sources, frequency of collection, and a rationale for each proposed indicator.
This post is part of a six-day series reflecting on lessons learned, highlighting best practices, and sharing recommendations from ‘Evaluation Without Borders’ (EWB), the Pro Bono Evaluation program of Washington Evaluators, a Washington D.C. area affiliate of the American Evaluation Association.
The American Evaluation Association is celebrating Washington Evaluators (WE) Affiliate Week. The contributions all this week to aea365 come from WE Affiliate members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.