Welcome to the Evaluation 2013 Conference Local Arrangements Working Group (LAWG) week on aea365.Howdy! My name is Jennifer Hamilton, and I am a Senior Study Director at Westat and a Board member of the Eastern Evaluation Research Society, an AEA affiliate. I am also a statistician and methodologist, who sometimes tweets about evaluation, in addition to other things too embarrassing and geeky to mention here.
We have known for a while that the evaluation pendulum was swinging towards randomized designs, largely due to the influence of the Institute for Education Sciences (IES) at the U.S. Department of Education (DoE). IES has done this largely through leveraging its $200 million dollar budget to prioritize evaluations that allow impact estimates to be causally attributed to a program or policy.
Some evaluators have welcomed this shift toward experimental designs, while others have railed against it. Love it or hate it, I think the Randomized Controlled Trial (RCT) is here to stay. I say this with some conviction, based on my own experiences working with DoE and the fact that other federal agencies seem to be moving in the same direction. A case in point is last year’s memo from the Office of Management and Budget, (cleverly dubbed the OMG OMB memo). It asks the entire Executive Branch to implement strategies to support evaluations using randomized designs. For example, when applying for grants, districts could be required to submit schools in pairs, so that one could be randomly assigned to the treatment and the other to a control condition.
Even though I believe the field is benefiting from the increased focus on experimental designs, the bottom line is that they are still not appropriate in all (or even most) situations. A program in its early stages of development asking formative questions should not be evaluated with an experimental design. Moreover, it is often costly and difficult to implement a high quality RCT (and don’t even talk to me about trying to recruit for them). Lastly, experimental methodology focuses on obtaining a high degree of internal validity, which often means that you are limiting the degree to which you can generalize your results and reducing external validity.
- If you decide to utilize an experimental design, familiarize yourself with the What Works Clearinghouse (WWC) standards and procedures. Although getting their Good Housekeeping stamp of approval may not be your goal, the WWC has had a lot of *really* smart people thinking about methodology for a long time. If you follow their guidelines, you reap the benefit of their brain trust.
Hot Tips—Insider’s advice for Evaluation 2013 in DC:
We’re thinking forward to October and the Evaluation 2013 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). AEA is accepting proposals to present at Evaluation 2013 through until March 15 via the conference website. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.