I’m Allan Porowski, a Principal Associate at Abt Associates and a fan of experiments – when they’re conducted under the right circumstances. Experiments, commonly referred to as RCTs (randomized controlled trials) go through three stages: (1) crazy start-up period, (2) normal data collection period, and (3) crazy analysis period.
Hot Tips: Here are some tips to make that start-up period less crazy:
- Don’t Fall in Love with the Method: Too often, evaluators try to force a given method to fit reality instead of using it to measure reality. Even though we may really want to conduct an RCT, it may not be appropriate. Experiments are not appropriate for new initiatives because they may not yet have excess demand for services, necessary data collection infrastructure, an randomization-accommodating intake process, or staff buy-in. If these criteria are not met, then the program is not ready to be tested experimentally.
- Be Forward-Looking by Working Backwards: There’s no substitute for in-person planning sessions to hammer out evaluation details. A half day (or better yet, a full day) on-site is needed; and you’ll need a big whiteboard. It helps to start with a discussion of what the site hopes learn, and design the study to meet those goals. Starting out with the big-picture and moving into the details also gets the conversation off to a more productive start than diving into the nuances of randomization.
- Know Your Audience, and Let Them Know You: Don’t forget that when conducting an RCT, you are asking staff to replace professional judgment with a completely random process. That’s not an easy proposition to make. It’s really important to convey your understanding that RCTs can be disruptive, and explain what can be done to minimize that disruption. Likewise, teach program staff to think like an evaluator. Get them involved in formulating research questions, identifying mediators, and developing hypotheses about the relationship between program services to outcomes. Keep in mind that nodding does not equal understanding. RCTs are not intuitive to most people, including many researchers, so take the time to explain study procedures in multiple ways.
- Pressure-Test Your Sampling Frame: RCTs are often knocked for lacking generalizability, and unfortunately, that criticism is often warranted. Did you just recruit a bunch of sites that only serve left-handed kids in Boston? Recruitment is tough, but it’s even tougher to make a case that results are generalizable when your sampling frame doesn’t represent the program participants you’re studying.
Rad Resource: Key Items To Get Right When Conducting a Randomized Controlled Trial in Education. Though over 10 years old, the advice is timeless.
The American Evaluation Association is celebrating the Design & Analysis of Experiments TIG Week. The contributions all week come from Experiments TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hello Allan,
I see that you wrote about the in-person meetings being necessary now years later when the pandemic has struck and affected many face-to-face interactions and hope that there has been a way to navigate that in order for things to flow despite the difficult circumstances.