Hi. My name is Kristi Pettibone and I am a Research Scientist with The MayaTech Corporation and Manager of our Center for Community Prevention and Treatment Research. I have been helping Tom Chapel as the AEA Summer Institute Program Co-Chair this year.
A few days ago, Tom posted about being clear about the primary demand for a product or service (milk is good for you… Got milk?) and secondary demand (buy [brand] milk because it is locally farmed/cheap/vitamin-reinforced etc), as it relates to the primary and secondary demand for evaluation services. I’d like to talk a bit about chocolate milk.
While I was at the AEA annual conference in Florida last year, I read an article in USA Today about a group of 5th graders in Illinois who were advocating to get chocolate milk back onto the menu in their school district. The school district had dropped the chocolate milk product as an obesity prevention strategy. They soon found that the unintended consequence of this obesity prevention policy was that the students were not drinking milk at all – although the article did not indicate what they were drinking instead. The school relented and offered chocolate milk on Fridays and was going to evaluate changes in milk consumption on Fridays.
Lessons learned: We know that strategies that change the environment in which decisions are made have great potential to affect public behavior. Going forward, we need to understand how to best evaluate these environmental strategies and their effect on outcomes. One of the challenges with evaluating environmental strategies is determining what changes to measure. In the chocolate milk situation, evaluating changes in the sale of chocolate milk may help us understand elementary students’ cafeteria purchases. But if the true concern is childhood obesity, what outcomes can we evaluate, given typical resource limitations?
This is just one example of an environmental strategy implemented to affect behavior change, but regardless of topic (health, education, arts, etc.), we as evaluators have to be clear in our understanding of what it is we are evaluating and what is feasible to measure. There are many publicly available datasets for those interested in evaluating environmental change strategies, especially policy.
Resource: State Health Policy section of the Kaiser Family Foundation’s Website
http://www.kff.org/statepolicy/index.cfm
Resource: CDC’s Nutrition and Physical Activity Legislative Database http://apps.nccd.cdc.gov/DNPALeg/index.asp
Resource: The National Cancer Institute’s State Cancer Legislative Database
http://www.scld-nci.net/index.cfml
Hot Tip: Are you interested in environmental change strategies and how to evaluate them? Check out the AEA/CDC Summer Evaluation Institute, June 14-16 http://www.eval.org/SummerInstitute10/default.asp, where I’ll be conducting a session on evaluating environmental change strategies. There are lots of other great sessions available too!
Kristi, thanks for this update on what seems to be a fascinating evaluation story. There has been a lot of discussion about problems in how evaluations of policies around school milk have been undertaken and reported.
Trevor Butterworth at http://www.stats.org from George Mason University has criticised the way an influential evaluation only examined changes in inputs not consumption and made claims about the impacts that didn’t consider the impact of switches to chocolate milk – http://www.stats.org/stories/2010/ny_school_milk_swindle_feb3_10.html
There was a discussion of on our blog of issues relating to the actual evaluation
http://genuineevaluation.com/critiquing-a-partial-evaluation-%E2%80%93-is-a-half-full-glass-better-or-worse-than-no-drink-at-all/ and how it was reported http://genuineevaluation.com/misreporting-evaluation-findings-example-1/
What is heartening in your example is that local authorities are committing to local evaluation of the impacts of changes in implementation. This is a piece of evidence-based policy and practice that is sometimes left out.