My name is Doug Koch, I am an assistant professor in the Department of Industrial and Engineering Technology at Southeast Missouri State University.
As a university faculty member I find it both ironic and concerning that faculty evaluation instruments often have some very basic issues. Being knee-deep in the promotion and tenure process, I thought it might be appropriate to offer a few tips to new faculty. One thing I have found after using and looking at different universities’ evaluations is that they are often poorly constructed instruments that don’t always aid in documenting your teaching effectiveness. Additionally, if you are not using the evaluation correctly, the instrument may not provide the correct data you need, or at the very least, you end up spending a lot of time explaining the data and why it yielded the results it did. The reliability and validity of student evaluations is an ongoing dispute. So, without even considering the actual soundness of the instruments from a testing theory perspective, there are some basic things you can do or look out for when you are evaluating your performance as a faculty member or other profession.
We currently use a departmental evaluation and also the IDEA Student Ratings of Instruction. Each of them has their own unique nuances and there are a few things I have run into. For the departmental evaluations, some questions were worded in a manner in which a strongly agree is positive and a few questions were negative. If the raw data is printed in a table or displayed in a graph, as below, it looks a little out of place. Even with bold explanations of the two drops in the graph, some reviewers wanted clarification.
Yes, this is easily resolved with reversing the assigned weights for the analysis. That raised flags for others.
Lessons Learned:
- Make sure the evaluations are consistent.
- I would recommend using two different evaluations when possible. We use both a department evaluation and the IDEA. Think of it as using alternative forms to address reliability or error measurement concerns. If there are large discrepancies between the two, you can justify why.
- For the IDEA and similar evaluations, make sure you study how the evaluation works and how you need to use it. You select the objectives that you consider essential, as selecting too many objectives can have ill effects on your scores (see this video).
- Review the instructions on the IDEA website and meet with the group at your institution responsible for the assessment in order to be sure that you are getting the results that you can use to improve your teaching.
- Inform your students of the objectives of the class throughout so that they realize what they are evaluating you on.