We are Mansoor A.F. Kazi and Yeongbin Kim (University at Albany), realist evaluators for the Substance Abuse and Mental Health Services Administration (SAMHSA) System of Care Expansion grant at Chautauqua County, New York.
We use research methods drawn from both epidemiology and effective research traditions in partnership with human service agencies to investigate what programs of intervention work and for whom.
Lesson Learned: The emphasis is on data naturally drawn from practice, and therefore quasi-experimental designs can be used with demographic variables to match intervention and non-intervention groups. Binary logistic regression can be used as part of epidemiologic evidence based on association, environmental equivalence, and population equivalence. In this way, evaluators and agencies can make the best use of the available data to inform practice.
The realist evaluation paradigm focuses on investigating how interventions may work and in what circumstances. This approach essentially involves the systematic analysis of data on:
- Service users’ circumstances (e.g., demographic characteristics);
- Dosage, duration and frequency of each intervention in relation to each user;
- Repeated use of reliable outcome measures with each service user.
Hot Tip: Realist evaluators work in partnership with human service agencies to clean data and undertake data analysis with them at regular intervals and not just at the end of the year. This way, evaluators and human service agencies can work together to evaluate the impact of interventions on the desired outcomes utilizing innovative methods and addressing issues relevant for practice including diversity, investigating where and with whom the interventions are more or less effective, in real time.
As the data mining includes all service users (e.g. all students within a school district), it is possible to investigate the differences in outcomes between intervention and non-intervention groups, and these groups can be matched using the demographic and contextual data. Binary logistic regression can be used to investigate which interventions work and in what circumstances. The variables that may be influencing the outcome can be identified through bivariate analysis and then entered in a forward-conditional model. The variables that are actually influencing the outcome are retained in the equation, and those that are significant provide an exponential beta indicating the odds of the intervention achieving the outcome where the significant factor(s) may be present. This approach is used extensively in Chautauqua County, New York, including school districts and human service agencies, and the county (Chautauqua Tapestry) received the Gold Award for Outstanding Local Evaluation in 2010 from the federal agency SAMHSA.
The American Evaluation Association is celebrating SW TIG Week with our colleagues in the Social Work Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hello,
My name is Justin Garrison, I am currently learning about program evaluation. I enjoyed how you talked about a quasi-experiment and what information you can get from it and what it can be used for. I am a student still learning a lot and that Is a good piece of information I might use in the future.