Greetings, and welcome to a week’s worth of insights sponsored by the Design and Analysis of Experiments TIG! We are Laura Peck and Steve Bell, program evaluators with Abt Associates.
When deciding how to invest in social programs, policymakers and program managers increasingly ask for evidence of effectiveness. A strong method for measuring a program’s impact is an experimental evaluation, dividing eligible program applicants into groups at random: a “treatment group” that gets the intervention and a “control group” that does not. In such a design, when different outcomes emerge it can be interpreted as a consequence of the intervention. In this week-long blog, we examine concerns about social experiments, starting with ethics.
A common concern in planning experimental evaluations is the ethics of randomizing access to government services. Are the individuals who “lose the government lottery” and enter the control group disadvantaged unfairly or unethically? Randomizing who gets served is just one way to ration access to a funding-constrained program. Giving all deserving applicants an equal chance through a lottery, is the fairest, most ethical way to proceed when not all can be served. Furthermore, the good news is that program staff are wonderfully creative in blending local procedures with randomization in order to ensure they are serving their target populations and preserve the experiment’s integrity. For example, an ongoing evaluation of a homeless youth program lets program staff use their existing needs-assessment tools to prioritize youth for program entry while overlaying the randomization process on those preferences: it’s a win-win arrangement!
Even should control group members be disadvantaged in a particular instance, there is reason this might not be unethical (see Blustein). Society, which benefits from accurate information about program effectiveness, may be justified in allowing some citizens to be disadvantaged in order to gather information to achieve wider benefits for many. Society regularly disadvantages individuals based on government policy decisions undertaken for non-research reasons (for example, opening high-occupancy vehicle lanes that disadvantage solo commuters to the benefit of carpoolers). Unlike control group exclusions, those decisions are permanent not temporary.
Moreover, in a world of scarce resources, it is unethical to continue to operate ineffective programs. From this alternative perspective, it is unethical not to use rigorous impact evaluation to provide strong evidence to guide spending decisions.
Finally, social experiments are in widespread use, signaling that society has already judged them to be ethically acceptable. The ethics of experiments can be somewhat challenging in particular evaluation environments, but our experience suggests that ethics generally need not be an obstacle to their use.
Up for discussion tomorrow is what experiments can tell us about program effects, when researchers apply conventional and new analytic methods to experimental data.
Rad Resource:
For additional detail on the ethics question, as well as other issues that this week-long blog considers, please read On the Feasibility of Extending Social Experiments to Wider Applications.
The American Evaluation Association is celebrating the Design & Analysis of Experiments TIG Week. The contributions all week come from Experiments TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Dear Paul, Thank you for your comment. Indeed, regression discontinuity design is a solid alternative (see Keith Zvoch’s recent AEA365 post on the topic: http://aea365.org/blog/experiments-tig-week-keith-zvoch-on-strong-program-evaluation-design-alternatives/). The variable that involves the cut-point, which is at the center of the RD design, is one that should be unknown to program applicants and also not manipulable. I am not familiar with any RD applications that use a “deservingness” scale for that variable, but I am familiar with ones that use a “neediness” scale: for example Keith’s work on a summer program for struggling readers used reading performance to gauge the need for access to the program and then used a cut-point along that variable to craft a RD-based treatment group. When such a variable exists, RD designs are a strong alternative to randomization. Thanks again, Laura
Laura and Steve – Thank you for sharing your thoughts on the use of “experimental evaluation.” Working in higher education, we have an Institutional Review Board that would review a randomized control trial (RCT) and/or an “experimental evaluation” and assess the ethics around such a plan. Are there any external reviewers available to private and non-profit entities in assessing ethics around RCTs? The widespread use of social experiments in the U.S. are often conducted within health/medical, science, and educational institutions that have IRB boards. Stating that that the use of social experiments in the U.S. paves the way for experimental evaluation does not take into account all the checks and balances in place to ensure these are done ethically. Should experimental evaluation be used within organizations that do not have some type of external review entity?
Dear Kate – Abt Associates has an IRB as do other research companies. That is, just because we are in the private sector does not mean we are exempt from ethics reviews. To my understanding, smaller companies or independent consultants can access private IRBs to help review their research plans. I would love to hear from researchers in those settings about their common practice. Rest assured for my work: those checks are in place! And I appreciate your concern. My best, Laura
Hi Laura and Steve,
Thanks for your post. I agree that randomized experiments can be done ethically. However, I would argue that in many contexts providing the services to the most deserving would be more ethical than a random process, and a regression discontinuity analysis can provide highly valid answers.