My name is Steven Holochwost, and for the last 8 years I have worked for WolfBrown, an arts and cultural research and management consultancy. In my role as Senior Researcher, I help design and implement evaluations of arts education programs for children.
Lessons Learned:
I imagine that the circumstances of my work are similar to those you encounter: an organization has implemented a program, and they would like to know whether it “works”. They have a group of children who are in the program, and (if you are lucky), they have a group of children who are not in the program (i.e., a comparison group). Rarely do we have the opportunity to assign children at random to these groups, but by employing the techniques of quasi-experimental designs (e.g., multivariate regression), we are able to work around these issues and offer conclusions, however laden with caveats.
But there is another, less obvious challenge to doing good evaluation work in these contexts. Often the children with whom we work are arranged in groups (e.g., classrooms), which means that our data is hierarchically-organized, or ‘nested’, in its structure. What does this mean? In statistical terms, it means the assumption that residuals will be independent and identically distributed, which is common to nearly all tests of inferential statistics, is violated. In practice, it means that you might conclude that a program is doing something even when it is not.
Hot Tips: So what can you do to avoid this?
- You can adjust your standard errors, but software to do this is not readily available.
- You can enter group membership as a series of categorical variables, but if there is meaningful information associated with groups (for example, if your groups are schools, some of which are public and some of which are private), teasing apart effects associated with that information is difficult.
- You can analyze your data with a hierarchical model, which is specifically designed to accommodate nested data structures!
Rad Resources:
- Here you can find information about training opportunities, many of which are offered as intensive, week-long courses.
- This website offers some readings about hierarchical modeling, as well as some fantastic online utilities that can be used to explore the results provided by most commonly-used statistics packages (e.g., SPSS, SAS, and STATA).
The American Evaluation Association is celebrating Arts, Culture, and Audiences (ACA) TIG Week. The contributions all week come from ACA members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.