Our names are Mwarumba Mwavita, Katye Perry, and Sarah Wilkey. We are faculty and evaluators in the Center for Educational Research and Evaluation at Oklahoma State University.
Higher education has constantly been engaged in the development, revamping, and implementation of programs. Often these changes result in reorganizations of existing programs and contribute to a dynamic and shifting ecology, which advances a need for evaluation to determine if outcomes are congruent or discrepant with intent. While some stakeholders are anxious about evaluation and its use, others are unaware of what and how it could be of benefit to them and show the overall impact the program has. Evaluating such a program requires evaluators to assume different roles and begin building evaluation capacity with program personnel. The challenge is HOW?
Hot Tip 1: Understand that you are engaging in a discussion about evaluation with those who may not understand evaluation.
Speak in a language that is not intimidating and is clear enough to explain what evaluation is. Introduce yourself and explain your role—do what you can do build rapport. Help those you are working with to understand that the goal of evaluation is to gather information that will lead to sound decision making, not to punish or find fault.
Hot Tip 2: Determine the unique contribution the service/program you are evaluating makes to the institution.
Let the program personnel know that you understand the university environment is dynamic, and their program may be in flux. Talk with stakeholders and program personnel to identify the goals of the program being evaluated; this will help you understand how the program fits into the university at large. Take time and care to look for discrepancies in words and actions. Understanding the difference between what program personnel and patrons say they do versus what they actually do. Determine the hierarchy/structure of the program, and ask yourself ‘Who is really in charge?’
Hot Tip 3: Determine what information the program personnel/stakeholders expect the evaluation to yield AND when they expect a final write up of findings.
Knowing what is expected of the evaluation will help you determine who needs to be on your evaluation team—be sure to include people with skills and expertise as needed about evaluation and the institution. Understand who the critical stakeholders of the program are and the role they play. This will also help you understand the best way to collect and present information.
Rad Resources: We have found the books Evaluative Inquiry for Learning in Organizations and Program Evaluation: Alternative Approaches and Practical Guidelines to be very helpful in our work.
The American Evaluation Association is celebrating Assessment in Higher Education (AHE) TIG Week. The contributions all this week to aea365 come from AHE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
I think evaluation in higher education has some unique challenges. Of course there are exceptions, but for the most part the ecology of the academy hasn’t changed in 200 years and there continues to be strong resistance to change to this day.
One of the challenges to doing evaluation in this environment is that these institutions are confused about their goals. Who is the customer? Colleges and universities say they are for students but they are designed for faculty. This makes it very difficult to evaluate programs and services. And many of the stakeholders, especially faculty with Ph.D.s think they do, or think they should, already know about evaluation. And, truth be told, don’t want to know how effective they are. They are safe within their departments and disciplines.
Competitive pressures are changing the landscape for colleges and universities and I think dramatic change is coming but that has only recently started. So, I hate to play the cynic, but I think it’s going to take much more than your “Hot Tips” to make evaluation a useful tool in improving higher education institutions.