Welcome to the Evaluation 2013 Conference Local Arrangements Working Group (LAWG) week on aea365. My name is Brad Keller, and I work at Westat, an employee-owned research organization in Rockville, MD.
For several years, I’ve worked on a contract with the U.S. Department of Education (ED), to help them obtain and analyze performance data from grantees. A lesson learned from this work is the need to balance a) our desire for high-quality analysis and b) the grantees’ time and ability to properly collect and report data.
Lesson learned – not everyone is an evaluator, but often “normal” people can do more evaluation than they think. For these programs, teachers and administrators often collected data at their school, and reported them to ED. Then, ED would aggregate those school-level data together. To allow for proper aggregation, each grantee had to collect and report the same data in the same manner as all of the other grantees.
My inclination was to develop the most meaningful indicators possible. But the grantees collecting and reporting these data are not evaluators. So, they were often as used to rigorous data collection and reporting as I was to teaching second graders (not at all).
Because of this, we needed to respect that the grantees often did not have the background required to implement a complex data collection, nor did they generally have the time.
Cool tricks – Balance high-quality analysis with grantee time and ability: In the end, we developed semi-complex performance indicators, which required grantees to report with a bit more rigor than they were used to. But, we also provided a great deal of technical assistance to help them learn the data-collection and –reporting steps. In addition, we even developed spreadsheets with embedded macros to help them devise random samples, when necessary.
So, my lesson learned was that non-evaluators (aka normal people) can do more complex data collection and reporting than they think, if you give them the proper tools and training. In addition, I learned that a less-rigorous evaluation, if properly implemented, is more valuable than a pristine evaluation plan that no one can follow.
Hot tip about DC – I hope that you are coming to the AEA conference in October. If so, be sure to register early. While in the Dupont Circle area, I recommend taking a short (25 minute) walk up Connecticut Avenue to the National Zoo. (You could also take the metro’s red line to either the Woodley Park-Zoo/Adams Morgan or the Cleveland Park station and walk a few blocks to the zoo, which is in between those two stations.) The zoo is free and the grounds are open from 6 AM until 8 PM, with the indoor exhibits open from 10 AM until 6 PM.
We’re thinking forward to October and the Evaluation 2013 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Registration is now open! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.
I love how you acknowledge the expertise of the teachers in your post. With such a strong focus on building others’ evaluation capacity these days, I’m afraid I sometimes ignore or minimize their existing “capacity”. Your post is a great reminder for me always to try to draw a line from their expertise to the evaluation task at hand. Thanks!
Thanks for a little background, Brad. I’ve worked on local evaluation teams for such programs as you describe. What the Department of Ed might consider is providing enough funding that an evaluator could be part of the process in addition to having the PI team to collect these data for use at the national level. These broad sweeps of local data can tell you a lot. A solid evaluation plan at the local level can tell you other things. Both are valuable. One, alone, doesn’t tell the whole story. The Department of Ed could greatly increase the impact of its successful projects if strong local evaluations could be funded and completed.