Hello! I am Jay Szkola, senior program analyst in the Strategy, Evaluation and Learning Division at Good Shepherd Services. As an internal evaluator, I have the opportunity not only to evaluate the programs in my portfolio, but also assist staff in using data in their day-to-day work with participants. This doesn’t have to involve a great deal of complexity, and can be as simple as developing tools that translate data into an accessible format.
Welcome to internal evaluation week, hosted by the Internal Evaluation TIG! Thanks to all of the internal evaluators who contributed to this week’s blogs and to Annie Gleason (Good Shepherd Services in New York City) for coordinating the work! This week is dedicated to sharing insights and tools from evaluators across the country (and Canada!) about how they help organizations to put data and evaluation findings into action. Blogs this week offer a broad range of suggestions for helping stakeholders to use data, drawing on technology, data visualization, communication and framing, and participatory methods.
All of the community-based youth justice programs at our agency use the Positive Youth Development Inventory as a tool to help evaluate our impact on key youth development constructs, such as friendship and future orientation. We use the PYDI to assess our participants at baseline, and gauge pre-to-post changes upon program completion. These insights are useful for program planning and advocacy.
I also wanted to make sure staff could use the results to inform their work with participants directly, but initially experienced limited success with this goal. One barrier was that while staff liked the survey, they were unsure how to translate individual survey responses into insights and action.
Cool Trick:
In response, I constructed a simple Excel sheet to help with this translation. Program staff enter in participant survey responses, and get out easy-to-read graphs showing how the participant scored in each construct. As staff enter the data, a table aggregates each response by construct, which then links to the graphs.
Program staff and leadership liked the tool so much that when the NYC Department of Probation adopted the PYDI as a measure, the Excel tool was shared and is now used by other youth justice programs in the city.
(click for larger images)
Lessons Learned:
Offer a guide. For those who do not have a high comfort level with data, colorful graphs alone will not bring about understanding. Included on the Excel tool is a brief, jargon-free definition, a sample question, and a list of questions for each construct on the survey.
Think about multiple uses to get maximum impact. Whatever the data source (surveys, administrative data, etc.), talk to staff about how they could use the data. Once staff in our youth justice programs began seeing the PYDI as something they could use in day-to-day practice, it became less of a task to be completed “for the evaluators” and instead, a shared project.
The American Evaluation Association is celebrating Internal Evaluation (IE) Topical Interest Group Week. The contributions all this week to aea365 come from our IE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hello Jay,
I am in the process of completing a Professional Master of Education at Queen’s University and am currently taking a course that focuses on program evaluation design. Your examination of how staff can transform evaluation findings into actionable items that will improve a program is very relevant to the program I am creating an evaluation design for. The direct and applicable content of your blog post was enjoyable to read.
Microsoft Excel is always an area in which I have struggled, so your “cool trick” of creating an Excel spread sheet to help translate survey responses into insights and action will undoubtedly be beneficial for my program evaluation design. The speed at which an individual can input data results into this spreadsheet and receive immediate feedback is phenomenal and will allow for immediate program adjustment. There are concerns that the needs of a program will change faster than the evaluation process can accommodate. The program for which I am creating an evaluation design is one that is on a limited timeline (a school year), so it is important to parse findings and make adjustments quickly. The Excel spreadsheet you created will allow evaluators and others to visualize data in a way that is clear, accessible, and understandable. This ease of use will ultimately allow an individual to immediately implement program improvements, which is beneficial as Chen (2005) notes that sooner is better with regard to program evaluation. The earlier data is able to evaluated, the easier it is for all involved in program evaluation to begin implementing feedback.
Thank you for your informative post,
Rachel
This is awesome and very helpful tips to work with program staff. Thanks Jay!!