We are Tayo Fabusuyi and Tori Hill, Lead Strategist and Research Scientist respectively of Numeritics, a research and consulting firm based in Pittsburgh, PA.
We conducted an evaluation of the Black Male Leadership Development Institute (BMLDI), a year-long program in Western Pennsylvania for high-school aged African American males. The BMLDI is designed to give participants access to Black male role models, provide opportunities for interaction within a supportive peer group, offer a challenging curriculum and equip the young men with leadership skills with a view towards positively impacting their perspectives and values.
Our evaluation strategy consisted of a mixed method, multi-phase approach with formative and summative components. In implementing the summative part of our strategy, we sought a framework robust enough to adequately capture how effective program activities were in achieving program goals, and to also provide insights on the structure and efficiency of those activities.
The framework that we employed was a modified form of Grove et.al’s EvaluLead Framework. The framework is premised on an open systems environment in which three interrelated forms of behavioral changes at the individual level are examined; “episodic,” “developmental,” and “transformative. These behavioral changes were analyzed using two forms of inquiry; “evidential,” or those measured using quantitative instruments and “evocative,” those assessed through qualitative tools.
This robust strategy has allowed us probe beyond program outputs to a more comprehensive framework that takes into consideration the broader influences that often affect program outcomes of this nature. The evaluation strategy also naturally lends itself to data triangulation, an attribute that helped reduce the risk of incorrect interpretations and strengthen the validity of our conclusions and recommendations made as regards program changes going forward.
Lesson Learned:
- Given the myriad of factors that may influence program outcomes, the evaluation of programs similar to the BMLDI program are best carried out in an open systems environment. This also guarantees that the evaluation process will be flexible enough to make provisions for exit ramps in the evaluation process and to capture unintended outcomes.
Hot Tips:
- An equally robust data gathering method is required to monitor progress made towards program goals and adequately capture program outcomes. We would recommend a 2 dimensional evaluation framework – evaluation type x data type.
- For a behavioral change evaluation, goals should be focused on contribution, not attribution. The emphasis should be to show that program activities aided in achieving outcomes rather than claiming that program activities caused the outcomes.
RAD Resources:
- PREST training resource: Mixed Research Methods
- Grove et al’s EvaluLEAD
The American Evaluation Association is celebrating Mixed Method Evaluation TIG Week. The contributions all week come from MME members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.