We are Dr. Kelly Skinner, an independent evaluation consultant, Assistant Professor at the School of Public Health and Health Systems at the University of Waterloo (UW), and program lead for the Master of Health Evaluation; and Dr. Sneha Shankar, a post-doctoral research fellow and measurement and methodology specialist at McGill University. Together, we have been developing a study protocol to redevelop and build validity evidence for the Knowledge Uptake and Utilization Tool (KUUT).
In 2007, we developed the KUUT to evaluate the implementation of knowledge into practice. Using current measurement perspectives or the unified validity theory, we examine the use of the KUUT by health organizations. As an evaluator, you may need to develop new measures or tools as part of your practice. You also might want to examine the validity of these tools and how they are used in practice but aren’t sure where to start or how to proceed. Maybe our example can help!
Lessons Learned
The current view of validity evaluates how a tool is interpreted and the claims that are made about the tool. Validity information provides evidence for how a tool operates in different environments and with different stakeholders. Therefore, building validity evidence is an ongoing process. Since the current view of validity continues to be underutilized, we offer an example to demonstrate its application. One important and rarely examined aspect of validity is response processes, or how individuals interact with the KUUT. Our protocol illustrates how to examine response processes to investigate what individuals are thinking as they are engaged with the tool. This evidence will be combined with user and psychometric evidence to help us understand how to score the KUUT and what the score means.
Hot Tip
According to The Standards for Educational and Psychological Testing (The Standards), validity is defined as “?the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests” (p.11). The Standards describes five sources of validity evidence that together contribute to the overall construct validity of a measure or the “whole of validity” evidence. The three sources of validity evidence that will be examined as the KUUT is redeveloped and tested include validity evidence based on: test content, response processes, and internal structure. We propose four phases below:
(Image by Melody Morton Ninomiya)
Rad Resources
Though the KUUT is undergoing revisions, it is a great resource to start thinking about how to assess knowledge uptake and utilization in your work. You can see our presentation from the Performance and Planning Exchange Symposium in 2019 to get more details about the larger project. The protocol study will soon be published in the Health Research Policy and Systems journal. If you would like a copy of the updated KUUT once it’s ready, or want to be added to our growing Community of Practice, send me an email at kskinner@uwaterloo.ca!
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.