My name is Xiaomei Song, a PhD Candidate in Assessment and Evaluation, Queen’s University, Kingston, Canada. In the past four years, I have been working on my doctoral degree and a part of my dissertation is to evaluate a large-scale high-stakes testing system from the stakeholder perspective. I have learned a number of lessons along this academic inquiry.
Lessons learned:
- Be prepared for questions. Get yourself prepared for the questions like: “what do test takers know about the test?” in the meetings and team work. Psychometricians and evaluators live in different worlds. For sure they use different mindsets to approach and interpret phenomena. Traditionally accepted conceptualizations of important terms in testing such as validity and fairness rest on the assumption of objectivity and neutrality. Well, that is not what evaluators believe in.
- Identify key informants and legitimate stakeholders. In the Encyclopedia of Evaluation, Jennifer Greene emphasizes that since not all stakeholders are involved in the whole process of the testing, it is important to identify key informants and legitimate stakeholders. In her article, From products to process: an ecological approach to bias detection, Janna Fox shares that some stakeholders such as test takers and instructors may discuss about their perceptions associated with test design, item development, administration, scoring, use, impact and consequences. However, Catriona Scott in her article, Stakeholder perceptions of test impact found that stakeholders, such as administrators and parents, may provide meaningful information only in limited aspects.
- Don’t take knowledge and experience for granted. Don’t take for granted about stakeholders’ knowledge and experience. They may be titled as classroom instructors, but they can also be program administrators, test administrators, item writers, or even test takers. Test stakeholders are often separated into three groups in testing standards, guidelines, and codes of practice: test developers, test users, and test takers (e.g., AERA, APA, & NCME, 1999; JCTP, 2004). Apparently, a clear delineation of these roles may not always exist in reality. Survey questionnaire and interview questions need to take this into consideration accordingly.
Resources:
- The American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education jointly developed Standards for educational and psychological testing.
- The Joint Committee on Testing Practices’ Code of Fair Testing Practices in Education that provides guidance in the following:
- Developing and selecting appropriate tests
- Administering and scoring tests
- Reporting and interpreting test results
- Informing test takers
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.