EdEval Week: Jade Caines on Survey Design

Hi, I’m Jade Caines. I have taught prekindergarten through college grades for over 10 years and have also worked on numerous evaluation studies in the southern, western, and eastern parts of the U.S. I recently received a Ph.D. in Educational Research, Measurement, and Evaluation. Currently, I am a postdoctoral research fellow at University of Pennsylvania, where my research area includes the validity of scales and instruments used within education evaluations.

Lessons Learned:

I have been in many situations where people wanted to know whether or not something “worked.” Folks would slap together some questions, put it on Survey Monkey, and get some people to respond. But since my involvement in the Educational Measurement field, I have learned so much about survey design. The most critical lesson I’ve learned in creating surveys used within education evaluations is the importance of defining exactly what it is that is being measured. Often I’ve had conversations with clients where the “what” is taken for granted. I have learned to spend significant time probing, asking pointed questions about what they really want to measure. Eventually, clients realize that they may not be so sure about the one “thing” they want to measure. Questions, Questions, and more Questions have helped in defining a construct (that “thing”) and then operationalizing it (creating situations where that “thing” is represented) for survey design. A construct can be defined as the theoretical object of our interest in the respondent. For example, character in students may be a construct that evaluators may want to measure as a part of a character evaluation grant.

Hot Tips:

  • Name the “thing” you want to measure (e.g. perseverance in the classroom).
  • Create a list of situations/experiences where that “thing” is represented (e.g. a student resubmits a writing assignment three times for a better grade). This would be a list of indicators, or evidence that a certain “thing” exists.
  • Create a list of situations/experiences where the opposite of that “thing” is represented (e.g. a student chooses not to submit an assignment, despite multiple deadline extensions from the teacher).
  • Then decide how to represent these situations that span a continuum of that “thing” on a measurement tool (e.g. a survey).

Cool Trick:

Draw a vertical line where the top of the line represents a high amount of that “thing” and the bottom represents a low amount. Try to create a survey that has several questions in the high, medium, and low sections of that line.

Rad Resource:

Constructing Measures: An Item Response Modeling Approach by Mark Wilson (2005). The first 2 chapters are the most relevant.

Hot Tip: Take a minute and thank a teacher this week!

The American Evaluation Association is celebrating Educational Evaluation Week with our colleagues in the PreK-12 Educational Evaluation AEA Topical Interest Group. The contributions all this week to aea365 come from our EdEval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

4 thoughts on “EdEval Week: Jade Caines on Survey Design”

  1. Hi Jade, you have such an impressive resume! I am a Kindergarten teacher in British Columbia, Canada and I am currently in grad school completing a Masters Degree in Literacy Education. It was refreshing to read your article on survey design.

    During my undergrad I remember helping many of business degree friends fill out surveys on Survey Monkey – they were short but tedious and I never really knew why I was filling them out, I just wanted to be a helpful friend. I think that it’s important that surveys have a meaning and that meaning is clearly identified.

    I don’t use surveys in my classroom but if I were to maybe I would measure “kindness in the classroom”. I would have to start by observing the students in my class to indicate situations of when students show kindness -sharing, helping a friend in the classroom, etc. Then, I would need to create a list of times that students are being unkind – leaving friends out, being mean to classmates, etc. Then I would have to find a way to measure this…perhaps on a graph to see how many times students are being kind and unkind? Does that sound right? Obviously, this wouldn’t happen all at once I would have to make it clear to student’s what kindness looks like, then show them several examples and point out things they are doing that are kind. Then, finally I would graph it so that students have a visual.

    I like your second Hot Tip, Thank you for all your heard work!

  2. Pingback: A Roundup of Survey Design Resources (Cross-Post with actionable data) | Sheila B Robinson

  3. Pingback: A Roundup of Survey Design Resources | actionable data

  4. Great to see a focus on validity and scale development including constructs. As someone who’s doctoral work centered on this topic, I have found that this is an often overlooked aspect in the world of evaluation and even research. I hope that this will spur a focus on the importance of carefully creating surveys or other questionnaires. Thanks for the tips and suggestions.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.