Hi, I’m Joseph E. Bauer, Director of Survey Research & Evaluation in the Statistics & Evaluation Center (SEC) for the American Cancer Society (ACS) in Atlanta, Georgia. I am in my eleventh year as an internal evaluator. I am the former Chair of the Organizational Learning and Evaluation Capacity Building (OL-ECB) TIG, and am currently on our Leadership Team.
Lesson Learned: Observational studies are a broad class of research that are numerous across multiple fields of study, from medical research to health care and health policy, to health promotions, and to social and behavioral research. Quite often this type of research is framed as ‘inferior’ or ‘weak’ as compared to randomized controlled trials (RCT’s). That’s because the data derived in these studies (convenience samples) pose a threat to the validity of statements made about causal inference, because it introduces selection bias (among other kinds of bias) into the treatment assignment. On balance, observational studies yield relatively ‘low grade evidence’ and lack the ability to make valid causal statements or to generalize findings to a wider population. However, this does not mean observational studies are worthless, as they can and do provide understanding and insight into the human condition, are generally easier to implement, generate results more quickly, are less expensive, and are useful for generating hypotheses to be followed up on with more rigorous study designs. Interestingly, these same kinds of biases and weaknesses can occur in RCT’s as well, especially ones that are designed more as efficacy studies. However, RCT’s can be designed as effectiveness studies (also called pragmatic studies (practical)) and control for those biases and weaknesses. So, one must be careful and thorough in thinking through study designs for your own research and/or evaluations. Every design has strengths and weaknesses – it is not an ‘either/ or’ problem. We are often trying to balance efficacy (does the treatment work?) with effectiveness (does the treatment work in the ‘real world’ for different kinds of people?). Where one chooses to calibrate that balance depends on a number of factors, including the philosophy of your design. Observational studies and RCT’s can be valuable. The key word is transparency.
Hot Tips: While this short piece will not answer all the questions you may have about observational studies or randomized controlled trials – it will hopefully lead you to address the larger issue of the need to be transparent in the reporting of the methods used in the conduct of research or evaluation. Refer to: STROBE Statement (Strengthening The Reporting of OBservational studies in Epidemiology) and CONSORT (CONsolidated Standards Of Reporting Trials).
Rad Resources: The American Evaluation Association’s Guiding Principles for Evaluators–which are intended to guide the professional practice for evaluators.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Yes, observational studies are great for answering many research questions in many situations. The choice of research design should always be driven by what type of design can best answer the questions at hand.
Also, any type of research design can be performed in a rigorous or non-rigorous way. You can’t tell the rigor of a study by the type of research design.