My name is Michelle Jay and I am an Assistant Professor at the University of South Carolina. I am an independent evaluator and also an evaluation consultant with Evaluation, Assessment and Policy Connections (EvAP) in the School of Education at UNC-Chapel Hill. Currently I serve with Rita O’Sullivan as Directors of AEA’s Graduate Education Diversity Internship (GEDI) program.
Lessons Learned: A few years ago, EvAP served as the external evaluators for a federally-funded Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) state-wide grant housed at University of North Carolina (UNC) General Administration. Part of our work involved assisting project coordinators in 20 North Carolina counties to collect student-level data required for their Annual Performance Review reports as well as for program monitoring, assessment, and improvement. For various reasons, project coordinators experienced numerous difficulties in obtaining the necessary data from their Student Information Management Systems (SIMS) administrators at both the school and district levels. As collaborative evaluators, we viewed the SIMS administrators not only as “keepers of the keys” to the “data kingdom,” but also as potentially vested program stakeholders whose input and “buy-in” had not yet been sought.
Consequently, in an effort to “think outside the box,” the EvAP team seized an opportunity to help foster better relationships between our program coordinators and their SIMS administrators. We discovered that the administrators often attended an annual conference each year for school personnel. The EvAP team sought permission to attend the conference where we sponsored a boxed luncheon for the SIMS administrators. During the lunch, we provided them with an overview of the GEAR UP program and its goals, described our role as the evaluators, and explained in detail how they could contribute to the success of their districts’ program by providing the important data needed by their district’s program coordinator.
The effects of the luncheon were immediate. Program coordinators who had previously experienced difficulty getting data had it on their desks later that week. Over the course of the year, the quality and quantity of the data the EvAP team obtained from the coordinators increased dramatically. We were extremely pleased that the collaborative evaluation strategies that guided our work had served us well in an unanticipated fashion.
Hot Tip: The data needs of the programs we serve as evaluators can sometimes seem daunting. In this case, we learned that fixing “the problem” was less a data-related matter that it was a “marketing” issue. SIMS administrators, and other keepers-of-the-data, have multiple responsibilities and are under tremendous pressure to serve multiple constituencies. Sometimes, getting their support and cooperation are merely a matter of making sure they are aware of your particular program, the kinds of data you require, and the frequency of your needs. Oh, and to know that they are appreciated doesn’t hurt either.
This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.
I agree with Leeann’s comments,
You mentioned that the luncheon had immediate effects on the relationship between administrators and program evaluators, do you think that this helped the program evaluators and admin “think out of the box” because the conferences is in a relaxed setting?
Hi Michelle,
I thoroughly enjoyed reading your post. I am currently taking a program inquiry and evaluation course and we are looking at collaborative evaluation in terms of how it works for and, in some case might, hinder the process and outcome of an evaluation. I am happy to hear that you were all able to come together to gather the information that was needed. I wonder how the evaluation might have continued if the administrators had not been co-operative, or if they never understood what information was needed?
I find that when I am working on a project, sometimes, it is difficult to step outside the lens that I am looking through. We get so focused on the “task”. In the experience that you shared, you and your team were able to take a step back and it turned out that building a relationship as well as open communication led to success!
To what extent do you thing the relationship between evaluators and stakeholders sets the foreground for a successful evaluation?
Thank you for sharing your experience. It is a humble reminder to take a step back, sometimes, and look at the big picture/ build those relationships.
LeeAnn