Hello, I’m Oscar Espinosa from Community Science . We recently evaluated the effectiveness of professional development programs in various sectors that seek to diversify their leadership or workforce to be more responsive to communities of color.
Hot Tips
- Specify what program effectiveness means–to all stakekholders! A program’s intended objectives are oftentimes skewed to the perspective of the funder. As an evaluator, you need to consider the various program stakeholders and determine what effectiveness looks like for each of them. To that end, sessions to develop program logic models should be held with the funder and separately with other program stakeholders. Vetting and reconciling the models is an essential step to establish a good foundation, before moving on to an evaluation design. Allocate enough time for this process as reaching consensus can be a laborious task.
- Capture participants’ accomplishments but don’t downplay challenges. Despite pressures from funders, who understandably want to highlight positive impacts, as an evaluator you have to identify unintended program consequences and areas for improvement. Data collection needs to focus on challenges participants experienced, including perceptions that activities were not tailored to people of color or their cultural or linguistic needs. Be prepared to have uncomfortable discussions about structural racism or equity issues. Doing this can lead to solid recommendations for program improvement.
- Numbers and stories are BOTH essential. We were interested in what brought participants to the program; their expectations as compared to their actual experience; and the influence the program had on them. We found that a combination of forced-response survey items and open-ended, semi-structured interviews before and after participants complete the program were effective methods for getting a full picture.
Lesson Learned: To effectively evaluate professional development programs, one needs to take into account both funding organizations’ policies and culture and people’s of color needs and background. The evaluators’ art is their ability to extract the voice of program participants from the noise produced by program requirements and the institutional context. Ultimately, a program’s effectiveness should be judged on the extent to which is motivates people of color to continue to take on new challenges and advance in their profession.
Rad Resources
- Handbook on Leadership Development Evaluation is a comprehensive resource filled with examples, tools, and the most innovative approaches to evaluate leadership development in a variety of settings.
- L. Kirkpatrick’s Evaluating Training Programs focuses on evaluation approaches to measuring reaction, learning, behavior, and results.
- Special Issue: Building a New Generation of Culturally Responsive Evaluators through AEA’s Graduate Education Diversity Internship Program.
The American Evaluation Association is celebrating APC TIG Week with our colleagues in the Advocacy and Policy Change Topical Interest Group. The contributions all this week to aea365 come from our AP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.