Greetings. My name is Ricardo Gomez and I currently work as a Research and Evaluation Associate for the National Collegiate Inventors and Innovators Alliance. I am also a doctoral candidate in International Education at the University of Massachusetts Amherst, Center for International Education, and alumnus of the AEA-Duquesne University Graduate Diversity Internship Program.
The opinions of stakeholders are crucial because they can shape the direction of programs and can have an impact on program execution, scalability, and performance. Hence, as a researcher and evaluator, I have always been interested in finding ways to gauge the subjectivity (i.e., opinions, perceptions, attitudes, and motivations) of evaluation participants, and incorporate these into the different phases of my evaluation activities.
Lesson Learned – Q methodology is a powerful tool that evaluators can use to explore the perspectives of evaluation participants. First used and advanced by William Stephenson in the 1930s, Q is a research method that statistically identifies different points of view (or subjectivities) on a given topic based on how individuals sort a set of statements, about that topic.
Traditionally, evaluators have relied on interviews or surveys with Likert-type items to gauge the opinions of evaluation participants. These approaches are not without their drawbacks: the typical outcome of the analysis of Likert-type items is a description of pre-specified independent categories deemed relevant by the evaluator; and interviews can be time consuming and intrusive.
The outcome of a Q study, on the other hand, is a more authentic set of factors that capture people’s attitudes and perspectives about an issue. In Q method, a group of participants (the p-set), sort a sample of items (the q-set), into a subjectively meaningful pattern (the q-sort). Resulting q-sorts are analysed using correlation and factor analysis (q-analysis), yielding a set of factors whose interpretation reveals a set of points-of-view (the f-set).
Rad Resource: Click here www.broaderimpacts.org/aea2011 for an online example of a Q-sort process.
Lesson Learned: Q methodology is an important bridge between qualitative and quantitative methods in that it provides a means for analyzing the phenomenological world of a small number of individuals without sacrificing the power of statistical analysis.
Rad Resource – The International Society for the Scientific Study of Subjectivity (ISSSS) is the official organization committed to the ideas and concepts of Q methodology as enunciated by William Stephenson. ISSSS administers an email discussion list dedicated to exchange of information related to Q Methodology. To learn more about Q methodology, join ISSSS, or become a member of the email discussion list, please visit www.qmethod.org.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea3365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators
Hello, the “Rad Resource” link (www.broaderimpacts.org/aea2011)is not working for me. Is there another way to access the Q sort process example? Thank you!
I am confused…
a) Q-methodology can be visualized by SNA but its not SNA??? Am I correct???
b) If Q-methodology investigates individual experiences or perceptions of relationships…why this doesn’t mean that it investigates ties…between individuals and how they use resources-especially if it can be visualized by SNA???
c) Could you please elaborate a little more on “complimentary” instead of “alternative” role /way of thinking of Q-SNA relationship?
Many Thanks
a) Q-methodology can be visualized by SNA but its not SNA??? Am I correct???
answer: Q methodology is not SNA and cannot be visualized by SNA.
b) If Q-methodology investigates individual experiences or perceptions of relationships…why this doesn’t mean that it investigates ties…between individuals and how they use resources-especially if it can be visualized by SNA???
answer: Q methodology is used to investigate people’s perceptions or opinions on an issue. For example how a group of youth in the Middle East understand the concept of democracy; or what a group of program clients think about the delivery of services. Q has been used in many fields including, political science, education, evaluation, public health, environmental studies, marketing, etc.
SNA analysis, on the other hand, is used to examine and describe human relations, and the structure of communities.
c) Could you please elaborate a little more on “complimentary” instead of “alternative” role /way of thinking of Q-SNA relationship?
If you want to identify important actors, crucial links, subgroups, roles, network characteristics, SNA might be the way to go.
If you want to investigate what people think about those subgroups, roles, network characteristics, Q methodology might help.
Agree absolutely!
Pingback: Melissa McGuire on Selecting Outcome Indicators for Evaluation – Leverage Divergent Points of View with Group Priority Sort · AEA365
Thanks for sharing Ricardo. Q methodology is definitely an intersting research tool with a lot of potential. We recently published an article in Healthcare Quarterly, on a variation of the Q methodology, we call Priority Sort. Used to capture subjective opinions in group settings, this method results in both quantitative rakings and rich qualitative data. We’ve used it successfully in a number of different fields (including education, healthcare and economic development), but the example in the article is within a healthcare setting. An abstract and access to the full article can be found here: http://www.longwoods.com/content/22651
Thank you Melissa for your comment. I’ll definitely get a copy of your paper. It sounds really interesting and I think other colleagues in the evaluation community might be interested in learning more about Priority Sort. Have you thought about writing an entry for AEA365?
Cheers.
PS: I did not realise when responding above that Q methodology involves _ranked_ groups of items. This might require some tweaking of the use of network analysis methods proposed above. For example, there are two types of network links involved: (a) items are linked by being placed in the same rank group, (b) clusters of items are linked to varying degrees, according to the difference in their rank position
Thanks, Rick, for your comment. Yes, the Q sort process involves ranking a set of items, usually from most agree to most disagree, or any other type of scale that helps the participant express their opinion on the item under consideration. Individual sorts are then correlated and factor analyzed, and the resulting factor arrays are interpreted.
The results of Q-sorts (and in fact the results of any kind of sorting exercise)can also be visualised by social network analysis software, in a way that makes results easier to understand by people who are not that knowledgeable about factor analysis.
Items that are placed in the same pile by a respondent can be seen as “connected” in network analysis terms. Items placed together in the same piles by multiple respondents can be seen as strongly connected. When many participants are involved in the same sorting exercise clusters of highly connected (i.e frequently associated) items can appear.
Participants can also be seen to be connected, by placing the same two items in the same piles.Some participants may do this more than others, and thus be more connected. These are more likely to be sharing a common interpretative framework (but this needs to be tested by comparing their descriptions of the piles in which they have both placed the same items)
For more on this use of network analysis perspectives and tools, see this page on my website:http://mande.co.uk/special-issues/participatory-aggregation-of-qualitative-information-paqi/
regards,rick davies
We are looking to evaluate science teacher leader’s use of different resources in their social network (e.g. science coach, University scientists, other science teachers, etc.) What is you opinion of using Q-sort methodology as an alternative to traditional social network analysis or in conjunction with traditional means?
Hello Ken,
No, I don’t see it as an alternative to SNA, because they both serve different purposes. They complement each other, though. SNA will help you investigate the ties between individuals and how they use resources. However, you could also use Q to investigate those individuals’ experiences or perceptions of relationships.