Greetings, and welcome to a week’s worth of insights sponsored by the Design and Analysis of Experiments TIG! We are Laura Peck and Steve Bell, program evaluators with Abt Associates.
When deciding how to invest in social programs, policymakers and program managers increasingly ask for evidence of effectiveness. A strong method for measuring a program’s impact is an experimental evaluation, dividing eligible program applicants into groups at random: a “treatment group” that gets the intervention and a “control group” that does not. In such a design, when different outcomes emerge it can be interpreted as a consequence of the intervention. In this week-long blog, we examine concerns about social experiments, starting with ethics.
A common concern in planning experimental evaluations is the ethics of randomizing access to government services. Are the individuals who “lose the government lottery” and enter the control group disadvantaged unfairly or unethically? Randomizing who gets served is just one way to ration access to a funding-constrained program. Giving all deserving applicants an equal chance through a lottery, is the fairest, most ethical way to proceed when not all can be served. Furthermore, the good news is that program staff are wonderfully creative in blending local procedures with randomization in order to ensure they are serving their target populations and preserve the experiment’s integrity. For example, an ongoing evaluation of a homeless youth program lets program staff use their existing needs-assessment tools to prioritize youth for program entry while overlaying the randomization process on those preferences: it’s a win-win arrangement!
Even should control group members be disadvantaged in a particular instance, there is reason this might not be unethical (see Blustein). Society, which benefits from accurate information about program effectiveness, may be justified in allowing some citizens to be disadvantaged in order to gather information to achieve wider benefits for many. Society regularly disadvantages individuals based on government policy decisions undertaken for non-research reasons (for example, opening high-occupancy vehicle lanes that disadvantage solo commuters to the benefit of carpoolers). Unlike control group exclusions, those decisions are permanent not temporary.
Moreover, in a world of scarce resources, it is unethical to continue to operate ineffective programs. From this alternative perspective, it is unethical not to use rigorous impact evaluation to provide strong evidence to guide spending decisions.
Finally, social experiments are in widespread use, signaling that society has already judged them to be ethically acceptable. The ethics of experiments can be somewhat challenging in particular evaluation environments, but our experience suggests that ethics generally need not be an obstacle to their use.
Up for discussion tomorrow is what experiments can tell us about program effects, when researchers apply conventional and new analytic methods to experimental data.
For additional detail on the ethics question, as well as other issues that this week-long blog considers, please read On the Feasibility of Extending Social Experiments to Wider Applications.
The American Evaluation Association is celebrating the Design & Analysis of Experiments TIG Week. The contributions all week come from Experiments TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.