Susan Lowes on Pile Sorts

Hello colleagues. My name is Susan Lowes and I am currently Director of Research and Evaluation at the Institute for Learning Technologies at Teachers College, Columbia University. One of my challenges is to make the evaluation process interesting to those being evaluated so I am always searching for assessments that go beyond surveys and interviews.

Pile sorts, also called card sorts, are used to elicit participants’ understanding of a domain by observing how they group items in that domain. Pile sorts are engaging for participants while revealing information not easily available through direct questioning.

Hot Tip: Begin an interview with a card sort. All you need is a set of cards with words or pictures that you ask the participants to sort into piles of similar items. As they do this, you often ask them to explain their reasoning. Card sorts get people talking, work well with children or adults, and do not depend on facility with English.

Rad Resource:

An easily accessible article that describes how to get at cultural categories by using pile sorts is John B. Gatewood, “Culture … One Step at a Time,” online at http://www.lehigh.edu/%7Ejbg1/cogmeth.htm. Gatewood’s domain is fish, but you can substitute a domain that fits your own work.

Hot Tip: Pile sorts have been used in many different disciplines—for example, by social psychologists to study racial stereotyping, by anthropologists to study social class, by information architects in usability studies, and to distinguish experts from novices in a number of different domains, and by ourselves to explore students’ understanding of who is and is not an engineer, their understanding of sensors, and gendered perceptions of video games.

Rad Resource:

Another interesting article is Gun Roos, “Pile Sorting: Kids Like Candy.” She is looking at children’s perceptions of different types of food. It is in Victor D. de Munck and Elisa J. Sobo, eds., Using Methods in the Field: A Practical Introduction and Casebook (Walnut Creek, CA: Altamira Press, 1998).

Hot Tip: Pile sorts are amenable to simple or sophisticated statistical analysis, including various types of multivariate analyses.

Rad Resource: G. Rugg and P. McGeorge’s article, “The sorting techniques: a tutorial paper on card sorts, picture sorts and item sorts,” is a good introduction to the different methods of analysis. It is in Expert Systems, vol. 14, no. 2 (May 1997): 80-93.

Hot Tip: In our work, we have developed an online version of the pile sort that we have found to be even more engaging than the paper version, while providing more easily accessible data for both pre-post and process analysis.

Rad Resource: Come to our workshop at the AEA meetings to learn how the digital version works.

This contribution is from the aea365 Tip-a-Day Alerts, by and for evaluators, from theAmerican Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org. Want to learn more from Susan? She’ll be presenting as part of the Evaluation 2013 Conference Program, October 14-19 in Washington D.C.