I am Alice Hausman, a professor of Public Health at Temple University. I have been working as a community based participatory research (CBPR) evaluator of youth violence prevention initiatives in urban environments for many years.
Lesson Learned:
- Involve the Community in Identifying Measures and Data. As part of the participatory evaluation planning process, I always ask community participants to define their vision of program success. But I take it one step further by looking for data that might actually measure these community-defined outcomes. The process of working with community partners to identify measures and data has been as rewarding as just asking what success would look like.
Hot Tips
- Use available data sources in partnership with the community. One community collaborative I worked with identified available data sets and survey opportunities they could use to evaluate their programs. In another project, a randomized community trial of a multi-level violence prevention program, we found that the standardized psychometric tools being used by the evaluation trial could be used to measure community-defined constructs, such as “showing kids love”, after reconfiguring the items through a participatory review process.
- Remind yourself of the value of community-evaluator partnerships. In our case, the indicator itself was insightful about the community’s perception of social and relationship factors related to preventing youth violence. But the actual process of discussing the instruments and constructs was rewarding for all parties. The academic researchers learned more about the lived experience of their community partners who learned more about measurement development and psychometric research.
- Don’t hesitate to collaboratively develop new measures Another important outcome of the process of identifying existing data to measure community ideas was the realization that new measures and data might be needed to accurately capture the constructs defined by the community. While our community partners were initially concerned with the burden of adding new questionnaires, their views shifted somewhat after seeing that the benefit of being able to actually measure community defined constructs would outweigh the risks of more surveys.
Rad Resource:
Get Involved: I would love to hear from others who have done work in this area. We can compare notes on indicators and measures and possibly find ways to make measuring community-defined outcomes as routine as measuring outcomes defined by funders.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hi Alice,
Thanks for the great post. It’s really good to see instances like this of participation-based indicator generation. You might find a paper I’ve co-written with Meg Satterthwaite on the challenges of generating meaningful human rights indicators to be of interest: “Measuring Human Rights: U.N. Indicators in Critical Perspective,” available at: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1916796
There’s some interesting work going on through the American Association for the Advancement of Science too, specifically their “Scientific Responsibility, Human Rights and Law Program” http://srhrl.aaas.org/
I’d be very interested in helping out with any initiative designed to, as you put it, “make measuring community-defined outcomes as routine as measuring outcomes defined by funders.” Do get in touch if you think I can be of assistance.
Anjie
Alice, it seems that what you’re doing is creating a “Line of Sight” diagram. I first learned about this concept from Marnie Mason, a well known quality improvement consultant. A community indicator is your long-term measure but the line of sight allows program/agency staff to construct a diagram about how what they do and what they measure in the short-term contributes to intermediate and longer-term outcomes. Here’s an example from public health: We inspect restaurants (# of inspections/process measure) SO THAT conditions in restaurants don’t create unsafe food (# of critical violations/outcome measure) SO THAT the public is sold food that is safe to eat (% of critical violations that are corrected within 24 hours/intermediate outcome) SO THAT there are fewer incidents of food borne illness (indicators: rates of food borne illnesses, a long-term outcome) that many different organizations and individuals within the community work towards achieving. This is similar to a logic model but much more concise and really focused on making sure that whatever each community member is working on they know how it contributes to those community-level indicators. This is also along the lines of the results-based accountability work of Mark Friedman (Los Angeles County Public Health uses his model) and so does the City of Bellingham and Whatcom County Health Department where I work. I’m currently taking the AEA eStudy offering on social network analysis as another important way to measure community partnership work. Thanks for your great post.