My name is Christine Paulsen and I own Concord Evaluation Group. We evaluate media and technology-based initiatives. We regularly integrate usability testing with program evaluation to provide our clients with a more comprehensive picture of how their technologies and initiatives are performing.
As evaluators, you know that many of the programs that we evaluate today are technology-based. It is not uncommon for initiatives to provide information to their target audiences via websites, while other interventions are delivered with software applications to mobile, handheld or other devices. To properly evaluate such initiatives, the evaluator must consider the usability (user-friendliness and accessibility) of the technology components.
Usability refers to how easily users can learn and use technology. It stands to reason that if a drop-out prevention program relies mostly on messages delivered via its website to change student behaviors, that website better be usable! So, as evaluators, it’s crucial that we include usability assessment in our evaluations. Usability testing (UT) methods enable us to not only gather important formative data on technological tools, UT methods also help us explain outcomes and impact during summative evaluation.
Hot Tip: Keep in mind that problems addressed early are much less expensive to fix than problems found later.
The typical UT is conducted in a one-on-one manner, with a researcher guiding the session. The participants are provided with a list of tasks, which they will likely complete while thinking aloud. The researcher will record both subjective comments as well as objective data (errors, time on task). The test plan documents methods and procedures, metrics to be captured, number and type of participants you are going to test, and what scenarios you will use. In developing UT test plans, evaluators should work closely with the client or technology developer to create a list of the top tasks users typically undertake when using the technology.
Hot Tip: Did you know that UT can be conducted in-person or remote (online)? While in-person testing offers a chance to observe non-verbal cues, remote testing is more affordable and offers the chance to observe a test participants in a more “authentic” environment—anywhere in the world.
Hot Tip: During formative testing, 6-8 users per homogenous subgroup will typically uncover most usability problems. The sample size will increase if inferential statistics are needed.
Rad Resource: For a great overview of usability testing, including templates and sample documents, visit Usability.gov.
Rad Resource: For a demonstration of how to integrate UT into your evaluation toolbox, please stop by to see my presentation at AEA 2011.
This contribution is from the aea365 Tip-a-Day Alerts, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org. Want to learn more from Christine? She’ll be presenting as part of the Evaluation 2011 Conference Program, November 2-5 in Anaheim, California.