Hello! We are Silvana Bialosiewicz and Kelly Murphy from Claremont Graduate University. Working as Senior Research Associates at the Claremont Evaluation Center, we have often been tasked with analyzing large quantitative databases and drawing conclusions about program effectiveness based on our results. Today, we’d like to share some hot tips about pesky sources of bias that may be lurking in your data and provide some rad resources about how to uncover this bias.
Hot Tip: Conduct tests of Measurement Invariance on your evaluation surveys
We all know that the accuracy of the self-report measures we use to assess multi-dimensional constructs (e.g., self-esteem, organizational commitment) in our evaluations is contingent upon the reliability and validity of our measures, but did you know that these measurement properties are not always generalizable to the different populations and program contexts we find in our evaluations? For example, have you ever wondered…
- If program participants from different cultures or socioeconomic backgrounds are interpreting your survey questions in the same way?
- If a participant’s gender, age, or literacy level affects the way they respond to your survey?
- If participating in the program changes the way participants think about your survey questions?
Answering questions such as these in a statistically rigorous manner helps us ensure that the comparisons we make (either across time or across groups) represent true differences in our constructs of interest!
Measurement Invariance:
What is it?
Measurement Invariance is the statistical property of a measurement that indicates that the same underlying construct is being measured across groups or across time.
How do we know if we have it?
When the relationship between manifest indicator variables (scale items, subscales, etc.) and the underlying construct are the same across groups or across time.
How do we test for it?
Because measurement invariance is too dense to cover in a single blog post, we have put together some rad resources to help you learn more about measurement invariance and the steps to assess measurement invariance.
Rad Resource #1: Based on our AEA13 demonstration session, we have put together an extensive resource packet for practitioners who are interested in learning more about measurement invariance and how to test for it.
Rad Resource #2: If you’re interested in learning more about the software used to assess measurement invariance, here is a link to a discussion thread on the strengths and weaknesses of available software.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.