My name is Stuart Henderson. I am the Associate Director of Evaluation for the Clinical and Translational Science Center at the University of California, Davis. We recently used a combination of screen recording software and think aloud methodology to conduct an evaluation of an innovative software program. I’d like to share how screen recording software and think aloud methodology might be useful in meeting other evaluation goals.
Screen recording software, as the name implies, is a program that records everything that is on a user’s computer screen. An example of this software is TechSmith’s CamtasiaStudio, http://www.techsmith.com/camtasia/, but this is just one of many screen recorders on the market.
Think aloud methodology, also referred to as verbal protocol analysis, is a technique where you have someone perform an activity or solve a problem and simultaneously verbally express their thoughts, feelings, and reactions as they are occurring. The theory is that by having subjects think aloud as they are doing something, you can better understand their cognitive processes and logic as they unfold. It is a common research technique in technology usability research as well as in some educational research.
Hot Tip: Many evaluators are turning to web-based surveys for their data collection needs, yet how our subjects are interpreting the questions or organization of our web surveys may be unclear. Conducting think alouds with screen recording software can be used to conduct cognitive interviewing of survey takers to help us understand how people are interpreting the questions and choosing their answers. These techniques also provide the opportunity to identify non-cognitive responses, so you can identify when your survey takers are frustrated, prideful, etc.—reactions that would be very difficult to capture through traditional methods.
Hot Tip: For evaluators who are creating databases or other programs for stakeholders and clients, think alouds and screen recordings might be a useful way to fine-tune these programs. We think we know how people are using the program, but until we watch someone use it and describe their reaction to it, we will be getting only part of the picture. Watching people use programs also allows us to identify active learning, for example, how people improve at using a program and begin to develop “work arounds” so that they get the program to do what they want.
Hot Tip: Screen recordings can also be used to share evaluation findings with stakeholders who are not local. With screen recording technology, it is easy to record your voice over PowerPoint slides or video and share the presentation with others to listen at their convenience.
Rad Resource: slides of our recent AEA talk on this topic can be found in the AEA elibrary. http://comm.eval.org/EVAL/model/Resources/ViewDocument/Default.aspx?DocumentKey=cd4acecd-ed34-4e49-9ec0-e185777e4e93
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.