NA TIG Week: Using a Needs Assessment Double Scale Survey Method to Pilot Test a Survey Instrument by Sue Hamann

I’m Sue Hamann, an evaluator for 40 years, and employed at the National Institutes of Health as a Health Scientist and Science Evaluation Officer.  Many people who are not trained in the analytic sciences and social research sciences are called upon to conduct evaluations.  For the last several years, I’ve tried to bring issues about needs assessment and survey development to novice evaluators.  I’m sharing an example of applying a needs assessment data collection method to pilot testing a survey.

Case. I was asked to pilot test an instrument designed to measure the impact of a new policy (National Institutes of Health Notice Number NOT-OD-15-103, Enhancing Reproducibility through Rigor and Transparency. Release Date: June 9, 2015) on improving rigor and transparency in biomedical research proposals (let’s refer to this as the Rigor Instrument).  The major activity in the policy evaluation plan was that participants would rate a series of actual research proposals submitted before and after the new policy was implemented to measure how much and in what specific ways proposals changed as a result of the new policy.

The Pilot Instrument was designed in a typical needs assessment fashion with a double scale survey, measuring two conditions of interest. The entire Rigor Instrument is included within the Pilot Instrument; that is, the elements or items in the center and the left-hand column ratings are the Rigor Instrument.  The Rigor Instrument contained about 100 items or elements to be rated.

The other technique that I used was a “thinking-aloud” protocol.  I met individually for 90 minutes with five pilot subjects, who were all accomplished researchers.  De-identified research proposals were available. After an introduction and review of instructions, I asked the pilot subject to read the Pilot Instrument aloud, starting with the written instructions, then the selected proposal, while also taking time to voice thoughts and impressions about the proposal, and then work through the instrument and the pilot test, vocalizing thoughts about the elements and the ratings.  All utterances were recorded by hand.

The pilot testing revealed that the elements or items were in general important and distinct.  It also revealed that numerous elements or items needed to be edited or reworded.  After making the indicated revisions, we were ready to start pre-testing the Rigor Instrument.

Lessons Learned:

• Methods and techniques typical of needs assessment can be applied to pilot testing an instrument.
• Instrument development follows an iterative process: test-revise-test-revise.
• Evaluators need experience and skill in survey design and development and quantitative and qualitative data analysis skills.

Rad Resources:

For Evaluation Planning:

• Jim Altschuld’s books address the design and utility of needs assessments surveys. 
  • Planning and conducting needs assessments: A practical guide. Witkin and Altschuld (1995).
  • From Needs Assessment to Action: Transforming Needs into Solution Strategies. Witkin and Altschuld (1999).
• Professor Herbert Simon (Nobel Prize winner, 1978, Economic Sciences) was a pioneer in treating text and verbalizations as data and in developing thinking-aloud protocols.
  • Protocol Analysis, Verbal Reports as Data, Revised. Ericsson and Simon H.  (1993).

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.