Hello, we’re Allison Watson, Peyton Holzworth (Fellows), and Sara Wolicki from the U.S. Centers for Disease Control and Prevention in the Division of Select Agents and Toxins (DSAT). DSAT oversees the Import Permit Program (IPP). IPP regulates the importation of infectious biological agents and vectors that could cause disease in humans to prevent their introduction and spread into the United States. To administer this program, DSAT created the electronic Import Permit Program (eIPP) information system. eIPP is a moderate security, cloud-based, electronic information system for receiving all import permit applications from U.S. importers.
The Office of Management and Budget (OMB) Circular A-11 requires that all capital investments, including information systems, complete annual operational analysis as part of the Management In-Use phase of the asset life cycle. Within its Capital Programming Guide supplement to the A-11, OMB lists three areas of consideration as part of the operational analysis activity, which includes seven questions that reveal useful information about each area. To complete this annual requirement, DSAT utilized the User Experience Questionnaire (UEQ), which is an evidence-based, validated questionnaire that measures a user’s experience with an interactive product. The UEQ focuses on hedonic and pragmatic qualities, which correlate to three constructs in each area (Image #1).
Rad Resource
Benefits of the UEQ:
- Open source (free)
- Available in 36 languages
- Generic and can be applied to a variety of interactive products
- Includes 8-prompt and 26-prompt versions of the questionnaire
- Website includes a turn-key user handbook, Excel spreadsheet for analysis, and publications explaining the theory and validation methods
- Provides a validated questionnaire addressing key objectives identified in the Capital Programming Guide, supplement to the OMB Circular A-11
First, we evaluated the eIPP information system users’ experience. Since we were already required to assess our users’ satisfaction for an annual operational analysis report, we decided to go above and beyond the requirement by combining the UEQ with the Key Performance Indicators (KPIs). KPIs accompany the UEQ and align with the UEQ’s qualities and constructs. Using both questionnaires, we assessed our eIPP information system users’ current experience with eIPP information system using the UEQ and compared it to their ideal experience using the KPIs.
Based on our users’ responses, we have an understanding of their key areas of interest and can generate ideas for system improvements. Overall, the results helped to support our use of the UEQ and KPIs to gather information for the annual operational analysis. We can also use the UEQ and KPIs longitudinally to view trends in our users’ experience over time. Additionally, the UEQ and KPIs are generic and can be applied to a variety of interactive systems, allowing us to compare improvements, features, and users’ experiences overall. We are excited about utilizing the UEQ and KPIs to evaluate our information systems. This open-source tool is great for government programs that may be looking to evaluate their own interactive systems.
_________
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
This report was supported in part by an appointment to the Research Participation Program at the Centers for Disease Control and Prevention administered by the Oak Ridge Institute for Science and Education through an interagency agreement between the U.S. Department of Energy and CDC.
The American Evaluation Association is hosting Government Evaluation TIG Week with our colleagues in the Government Evaluation Topical Interest Group. The contributions all this week to AEA365 come from our Gov’t Eval TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.