Greetings, I am Laura Pryor. In addition to being a GEDI alumna, I am a student at UC Berkeley’s Graduate School of Education in the Quantitative Methods and Evaluation program. As part of my graduate evaluation work, I have been exploring the recent trend of using multiple measures to evaluate teachers. As part of this trend, many policymakers and district leaders are combining multiple measures into a summative composite score, often for the purposes of high-stakes decision making (such as salary and personnel).
As a graduate student evaluator, I have been exploring two questions:
1) Is it necessary and/or purposeful to create a composite score?
2) If so, how should an evaluator combine multiple measures into a single composite score?
I hope this post provides insight into these questions so that evaluators can more easily navigate the increasingly popular context of high-stakes teacher evaluations.
Hot Tip 1: The purpose of the evaluation should decide if a composite score is needed. While it may be a current trend, not all multiple measure evaluation systems are used for a personnel or salary decision. For many districts and schools, the evaluation system is used to help teachers/staff identify areas for improvement; in this case, a composite score is not always necessary. If the evaluation system is intended for multiple purposes, prioritize purposes with stakeholders and discuss the feasibility for the evaluation system to embody multiple uses.
Hot Tip 2: If creating a composite score, select a model that is most appropriate for the evaluation:
a. The conjunctive approach: A pass/fail score is given; individuals must score at a specified passing level on every measure.
b. The disjunctive approach: A pass/fail score is given; individuals are only required to score at a passing level on one of the measures.
c. The compensatory approach: Individuals are given a continuum of scores; low scores on certain measures are compensated for by high scores on other measures.
Hot Tip 3: When using a compensatory approach, decide how to combine the measures:
a. Clinically: Evaluation stakeholders decide how to weight each measure; this is often called the ‘eyeballing’ approach.
b. Statistically: Select a criterion target and use regression methods to statistically determine the weights for each measure; this approach is considered more accurate than the clinical approach.
Rad Resources:
- For more on the purposes and design of a multiple measure teacher evaluation system, review The Personnel Evaluation Standards
- For more on the three compensatory models, see the chapter by Mehrens in: The New Handbook of Teacher Evaluation
- For a recent example of using a statistical approach to create a composite score, browse the reports under the Bill and Melinda Gates Measures of Effective Teaching Project
AEA is celebrating GSNE Week with our colleagues in the Graduate Student and New Evaluators AEA Topical Interest Group. The contributions all this week to aea365 come from our GSNE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.