AEA365 | A Tip-a-Day by and for Evaluators

May/12

11

EdEval Week: Jade Caines on Survey Design

Hi, I’m Jade Caines. I have taught prekindergarten through college grades for over 10 years and have also worked on numerous evaluation studies in the southern, western, and eastern parts of the U.S. I recently received a Ph.D. in Educational Research, Measurement, and Evaluation. Currently, I am a postdoctoral research fellow at University of Pennsylvania, where my research area includes the validity of scales and instruments used within education evaluations.

Lessons Learned:

I have been in many situations where people wanted to know whether or not something “worked.” Folks would slap together some questions, put it on Survey Monkey, and get some people to respond. But since my involvement in the Educational Measurement field, I have learned so much about survey design. The most critical lesson I’ve learned in creating surveys used within education evaluations is the importance of defining exactly what it is that is being measured. Often I’ve had conversations with clients where the “what” is taken for granted. I have learned to spend significant time probing, asking pointed questions about what they really want to measure. Eventually, clients realize that they may not be so sure about the one “thing” they want to measure. Questions, Questions, and more Questions have helped in defining a construct (that “thing”) and then operationalizing it (creating situations where that “thing” is represented) for survey design. A construct can be defined as the theoretical object of our interest in the respondent. For example, character in students may be a construct that evaluators may want to measure as a part of a character evaluation grant.

Hot Tips:

  • Name the “thing” you want to measure (e.g. perseverance in the classroom).
  • Create a list of situations/experiences where that “thing” is represented (e.g. a student resubmits a writing assignment three times for a better grade). This would be a list of indicators, or evidence that a certain “thing” exists.
  • Create a list of situations/experiences where the opposite of that “thing” is represented (e.g. a student chooses not to submit an assignment, despite multiple deadline extensions from the teacher).
  • Then decide how to represent these situations that span a continuum of that “thing” on a measurement tool (e.g. a survey).

Cool Trick:

Draw a vertical line where the top of the line represents a high amount of that “thing” and the bottom represents a low amount. Try to create a survey that has several questions in the high, medium, and low sections of that line.

Rad Resource:

Constructing Measures: An Item Response Modeling Approach by Mark Wilson (2005). The first 2 chapters are the most relevant.

Hot Tip: Take a minute and thank a teacher this week!

The American Evaluation Association is celebrating Educational Evaluation Week with our colleagues in the PreK-12 Educational Evaluation AEA Topical Interest Group. The contributions all this week to aea365 come from our EdEval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · · ·

3 comments

  • A Roundup of Survey Design Resources (Cross-Post with actionable data) | Sheila B Robinson · September 7, 2015 at 6:46 pm

    […] Caines blogged on survey design as part of EdEval week. She too stresses the importance of “defining exactly what it is that is […]

    Reply

  • A Roundup of Survey Design Resources | actionable data · September 7, 2015 at 1:43 pm

    […] Caines blogged on survey design as part of EdEval week. She too stresses the importance of “defining exactly what it is that is […]

    Reply

  • Pedro Paz · May 11, 2012 at 11:16 am

    Great to see a focus on validity and scale development including constructs. As someone who’s doctoral work centered on this topic, I have found that this is an often overlooked aspect in the world of evaluation and even research. I hope that this will spur a focus on the importance of carefully creating surveys or other questionnaires. Thanks for the tips and suggestions.

    Reply

Leave a Reply

<<

>>

Archives

To top