Welcome to the Evaluation 2013 Conference Local Arrangements Working Group (LAWG) week on aea365.Howdy! My name is Jennifer Hamilton, and I am a Senior Study Director at Westat and a Board member of the Eastern Evaluation Research Society, an AEA affiliate. I am also a statistician and methodologist, who sometimes tweets about evaluation, in addition to other things too embarrassing and geeky to mention here.
Lessons Learned:
We have known for a while that the evaluation pendulum was swinging towards randomized designs, largely due to the influence of the Institute for Education Sciences (IES) at the U.S. Department of Education (DoE). IES has done this largely through leveraging its $200 million dollar budget to prioritize evaluations that allow impact estimates to be causally attributed to a program or policy.
Some evaluators have welcomed this shift toward experimental designs, while others have railed against it. Love it or hate it, I think the Randomized Controlled Trial (RCT) is here to stay. I say this with some conviction, based on my own experiences working with DoE and the fact that other federal agencies seem to be moving in the same direction. A case in point is last year’s memo from the Office of Management and Budget, (cleverly dubbed the OMG OMB memo). It asks the entire Executive Branch to implement strategies to support evaluations using randomized designs. For example, when applying for grants, districts could be required to submit schools in pairs, so that one could be randomly assigned to the treatment and the other to a control condition.
Even though I believe the field is benefiting from the increased focus on experimental designs, the bottom line is that they are still not appropriate in all (or even most) situations. A program in its early stages of development asking formative questions should not be evaluated with an experimental design. Moreover, it is often costly and difficult to implement a high quality RCT (and don’t even talk to me about trying to recruit for them). Lastly, experimental methodology focuses on obtaining a high degree of internal validity, which often means that you are limiting the degree to which you can generalize your results and reducing external validity.
Rad Resource:
- If you decide to utilize an experimental design, familiarize yourself with the What Works Clearinghouse (WWC) standards and procedures. Although getting their Good Housekeeping stamp of approval may not be your goal, the WWC has had a lot of *really* smart people thinking about methodology for a long time. If you follow their guidelines, you reap the benefit of their brain trust.
Hot Tips—Insider’s advice for Evaluation 2013 in DC:
- If you meet Chris Lysy, say hi. If you are lucky, he may draw you.
- Go to Ben’s Chili Bowl.
We’re thinking forward to October and the Evaluation 2013 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). AEA is accepting proposals to present at Evaluation 2013 through until March 15 via the conference website. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.
In my opinion debates about the merits/shortcomings of RCTs are a misleading and ultimately unhelpful distraction. We waste a lot of time and effort on this debate to no useful outcome. The evaluation profession would be better served by debating the types of criteria we should be applying when seeking to make causal inferences and what counts as adequate evidence against these same criteria. Sound method choices must always be context dependent. The gold standard should be conclusions that are rigorous, credible and defensible; not the application of a particular technique. Building a consensus about evidentiary criteria will help us to rise above the RCT debate and move forward.
Agreed! I think folks like to make an argument dichotomous when in reality, there is plenty of middle ground. We can’t wish away RCTs, because they are a useful (dare I say, critical) tool in our methodological toolkit. But they are not our only tool (the hammer and nail analogy comes to mind). And as I pointed out, they should not be used mindlessly. There’s a time and a place for them, and it’s critical for evaluators to understand when and where that is.
Lot’s of good advice here, especially the Ben’s Chili Bowl recommendation. I cannot think of a better symbol for the diversity of Washington, DC than Ben’s Chili Bowl, owned by the late Ben Ali, a Muslim serving great chili in the historically African-American Shaw neighborhood to people of all backgrounds. As his biography notes, “Although Ben has moved on, his legacy continues with his family at the Chili Bowl helm. After surviving the 1968 riots and other challenges over the past 51 years, Ben has made sure that the Bowl will continue to serve its loyal customers for many years to come.” [Source: http://www.benschilibowl.com/ordereze/Content/14/Summary.aspx%5D.