I am Karen Zannini Bull, the Assistant Director of Distance Learning at Onondaga Community college and doctoral student at Syracuse University in Instructional Design, Development and Evaluation. My interests reside in Evaluation Theory and what comprises a good evaluation decision. With that in mind, I wanted to share lessons learned and a great resource for evaluators.
Lesson Learned: Many factors contribute to making a good, sound, quality evaluative decision. There is no single formula or recipe to conduct a successful evaluation. Each evaluation decision differs based on many factors including the goals, mission and vision of the organization, the stakeholders involved, the resources allocated (grant monies or otherwise), and the evaluators conducting the evaluation.
Sub Lesson Learned: Even though an evaluator may conduct many evaluations, this does not mean the evaluator will necessarily make the same decision twice. An evaluation conducted for a local school district yielded suggestions for ways in which the district could save thousands of dollars without cutting a single staff member. But would this same evaluation yield different results if the stakeholders had not emphasized the importance of retaining all staff members? Each situation is unique and each group of stakeholders varies. These factors influence change with regard to the variables and data considered when the evaluator makes a decision. With each variable variation, an alternate decision may be made.
Sub Lesson Learned: Never underestimate the power of outside forces such as time and cost. These two components can have a major impact on decision making. It is possible that an evaluator may make a different decision if given just one more week to collect data or two more days to look for themes across participants. What if an evaluator was given more funds to assist with manpower necessary for the given evaluation? Or what if, the client demanded results prior to the agreed upon deadline?
Sub Lesson Learned: Experience counts. If a novice evaluator and experienced evaluator conduct the same evaluation, with the same participants, the same context and stakeholders, side-by-side, the conclusions drawn in the end may be very different from one another. An experienced evaluator has vast knowledge of what works and doesn’t work, experience identifying themes that are subtle and an intuition about practicing evaluation.
Resource: Provided above was a synopsis of broader themes and issues that are apparent when conducting evaluations and making decisions. A great resource to discover more about evaluation practice and the nuances of decision making is Evaluation in Action: Interviews with expert evaluators by Fitzpatrick, Christie and Mark (SAGE, 2008). In short, “evaluation practice, as any professional practice, is concerned with subtleties, nuances, or larger shades of difference in how evaluators behave during a study” (p. 355).
Want to learn more from Karen or have a chat with her about her work? Attend the poster exhibition this November at Evaluation 2010!