AEA365 | A Tip-a-Day by and for Evaluators

Nov/15

27

DEOET TIG Week: Kim Manturuk on Increasing Survey Response Rates in Open Online Classes

Hello, my name is Kim Manturuk and I’m the Program Evaluator at Duke University’s Center for Instructional Technology. I get to evaluate a lot of interesting projects related to teaching and technology, but one of my biggest jobs is to evaluate Duke’s Massive Open Online Classes (MOOCs).

MOOCs are free, non-credit classes offered by universities and other institutions to anyone in the world. Since 2012, Duke has developed over 30 classes that have amassed over 2 million registrations. It’s my job to evaluate how well students in these classes are learning what the instructors want them to learn.

To accomplish this, I send out a lot of surveys – over 2 million surveys (and counting) in less than 3 years! When I started this project, I would be lucky to get 5% of people who registered for a class to fill out the pre-course survey, and the post-course survey response rates often hovered around 1%. It’s practically impossible to say anything evaluative with a 1% response rate, so I tried a lot of different things to get more people to fill out surveys. Some worked and some didn’t, but I learned several good lessons along the way.

Lesson Learned: When sending a survey to an online class, sign the email invitation with the class instructor’s name (with permission, of course). People are more likely to respond when the invitation comes from someone they know and respect.

Lesson Learned: Avoid sending surveys on Mondays when they are more likely to be ignored or accidently deleted during the first of the week email cleaning. It is better to send surveys from mid-morning on Tuesday through Friday.

Cool Trick: Tell people in the survey invitation how many questions they will be asked and how long it will take. I set it up so that students are asked just ten questions and then they are automatically brought back to their class.

Hot Tip: Be thoughtful about what demographic questions you ask in online classes. In some cultures, questions about race or gender are considered confusing, intrusive, or even offensive.

Rad Resource: Would you like to take a free class from Duke University? Visit our list of classes at https://www.coursera.org/duke. And if you do register for a class, be sure to fill out the course surveys!

The American Evaluation Association is celebrating DEOET TIG Week with our colleagues in the Distance Education and Other Educational Technologies Topical Interest Group. The contributions all this week to aea365 come from our DEOET TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

3 comments

  • katia El Kassem · November 30, 2017 at 12:56 am

    Hi Kim,
    I would like to Thank you for sharing those tips. however I have three questions for you.

    Question 1:

    Do you think learners were not responding to surveys because classes were non-credit and free?

    Question 2:

    have you seen any increase in the percent of people responding after trying the tips that you have stated ?

    to what extent those tips were beneficial ?

    Question 3:

    surveys were the online method to collect information on how well those students were leaning .

    To what extent do you think the evaluation would be valuable and measurable in such situation ?
    Thank you

    Reply

  • Megan From · November 25, 2017 at 2:46 pm

    Kim,

    Thank you for such practical tips about soliciting feedback for evaluation. It can be so difficult to get authentic feedback when, as you mentioned, only a small percentage of your stakeholders respond. As I think ahead to my future as an Administrator, your practical tips are indeed, very practical! I especially appreciate your tip about not sending your surveys at the beginning of the week! Monday morning unread emails can indeed be daunting so your suggestion of sending them after Tuesday at noon is brilliant. Thanks for sharing.

    Megan

    Reply

  • Rob Lippincott · November 30, 2015 at 1:57 pm

    Your tips on surveys are great! Sounds like you have captured much int he way of useful feedback….especially about whether — and what — MOOCs might be teaching effectively. Any “topline” findings you can share??

    Reply

Leave a Reply

<<

>>

Archives

To top