I’m Melissa McGuire, a Partner at Cathexis Consulting Inc., an evaluation practice in Toronto, Canada.
I was inspired by Ricardo Gomez’s recent post on Q methodology to share an adaptation of Q which we’ve been using in our practice that we call Group Priority Sort.
Lesson Learned: As many evaluators have experienced, it’s important to involve stakeholders in the selection of outcome indicators to get their buy-in and commitment. That said, we also know that consulting with stakeholders can stall the process, because different indicators are important to different groups. Worse still, sometimes organizations choose too many indicators in an attempt to appease everyone, which makes for a very cumbersome evaluation.
Rad Resource: Group Priority Sort is one method to rapidly get buy-in, build community, and find out which indicators are most important to measure, taking into account the perspectives of all stakeholder groups.
It is a hands-on exercise where stakeholders with different perspectives meet in person. The stakeholders are divided into small groups to sort the potential indicators (each printed on a cue card) along a continuum of relative “importance”.
Each group’s task is to determine the relative “importance” of all possible indicators through discussion and consensus building. The concrete outputs of Group Priority Sort are:
a) Consensus-based relative importance score for each indicator (for each group and averaged across groups).
b) Rich qualitative information about reasons given for relative importance of each indicator.
c) Engaged participants with a greater understanding of the perspectives of other stakeholders, greater understanding of the evaluation and greater ownership of the process.
The information obtained through Group Priority Sort can then be used to inform decisions about which indicators to ultimately include in your evaluation.
You can learn more about the other uses of Group Priority Sort and the methodology itself in this presentation or this article in Healthcare Quarterly.
Lesson Learned: Carrying out a Group Priority Sort can be resource intensive and requires careful planning and execution. However, done right, the method makes it possible to quickly narrow down priorities in a sound and meaningful way.
If you find opportunity to use Group Priority Sort, we’d love to hear about your experience.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Melissa, the link to the presentation PDF is broken. Would you mind re-linking it? Thank you for sharing this tool!
Hi Eva,
Here is the correct link: http://cathexisconsulting.ca/wp-content/uploads/2012/10/Priority-Sort_presentation-CES2010.pdf
I used a method combining this stakeholder-oriented indicator prioritization approach with another measure — investment required to collect indicator data; stakeholder input along with evaluator expertise helped to assess investment (time, resources, human labor) required to measure indicators and/ or proxies for indicators; we combined these weighted values to prioritize outcome indicators to measure for years 1, 2, and 3, as well as long-term. ongoing, follow-up indicators of a 3-year community food project. The method yielded awesome buy-in and increased evaluation capacity for stakeholders.
In my experience it is much easier to reach agreement on outcome indicators if one can first reach agreement about the program’s logic model. Discussions about outcome measures without first having an agreed logic model in place lack context and focus.