Hi! I’m Rachel Hickson, an evaluation specialist with Montgomery County Public Schools in Rockville, MD. Observations can be vital to measuring program fidelity. Yet they take a disproportionate share of effort and money when compared with other data collection methods. Can we stretch their utility to produce better data?
Lesson Learned: You don’t have a lot of time with those you observe, even with multiple observations. Plus, you need to mitigate the effects of “being observed,” since with observations, individuals know you are coming. Shahpar Modarresi and I conducted a large study on the relationship between teaching practices and student performance in mathematics. Here’s what we did to address these challenges.
- We asked our observed teachers to prepare a brief log about what they did before we arrived. When the appointment for each observation was made, we asked each teacher to complete a “lesson log” describing the four lessons leading up to the one we would observe. It improved our data in two ways: 1) It allowed us to collect more examples of what teachers were doing. 2) We could compare teachers’ self-reported practices and activities on their logs to observed practice, revealing differences or over-reporting of “desirable” practices. We used the same indicators of practice from our observation protocol to code what teachers included in their logs, so we did not have to develop additional codes.
Hot Tips: How else might you improve the utility of observation data?
- Ask participants for handouts. In our study about instructional practices, we asked teachers for a set of handouts for the observed lesson, as well as any related quizzes or homework These handouts confirmed the activities and topics we were seeing and improved the quality of our observations by giving us a basis for comparing assignments to what was taught.
The handouts served another important function. The collection of handouts became a data set for the district mathematics experts, who used them to help us analyze the scope and content of the material presented and confirm that it was within the parameters of the course being taught.
Overall, the triangulation of observations with self-reported practices and instructional artifacts led to better information with little additional effort.
Rad Resources: Information about our study can be found here in the AEA Public e-Library. Copies of our protocols can be found in the full technical report.
The CDC has a good primer on collecting observation data collection for program evaluation. Go to http://www.cdc.gov/healthyyouth/evaluation/data.htm and look for “Data Collection methods for Program Evaluation: Observation”
Another good overview on the observational method can be found here from the Forum for Qualitative Social Research.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.