My name is Judy Woods and I am a Doctoral student at Queen’s University in Kingston, Ontario, Canada. As a member of the graduate Assessment and Evaluation group, my first project as a novice evaluator involved the evaluation of an innovative medical education program. Being a Registered Nurse with extensive clinical experience in a university affiliated teaching hospital, and mentored by my supervisor as the Evaluation Consultant, this developmental evaluation in health care was an ideal orientation to the role of Principal Evaluator.
In this evaluation of a new educational program, stakeholders responded to questions asked in semistructured interviews. Content analysis of their collective responses were organized thematically and reported to program developers and implementers in the form of statement questions. Transforming findings statements into questions and then supporting these with program stakeholder interview responses will promote new discussion among program developers, administrators and implementers and contribute to further program decision-making. Presenting evaluation findings in this format, as developmental questions oriented towards program improvement, is a strategy with real potential for increasing evaluation product utilization.
Cool Tricks:
Step 1. Create the findings statement.
Step 2. Transform the statement into a question.
Step 3. Use stakeholder responses to describe, to extend understanding, or to make connections.
Formulating evaluation findings as developmental questions is a reporting strategy that aligns with program evaluation utility standards, in particular U5 Relevant Information and U6 Meaningful Processes and Products (Yarbrough, Shulha, Hopson, & Caruthers, 2011). Presenting findings as questions is a utilization-focused approach (Patton, 2011) intended to inform and encourage improvement-oriented discussion among program decision-makers. This reporting strategy is valuable in a developmental evaluation. Decision-maker responses to the questions then determine next steps in the program’s development.
References
Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2010). The Program Evaluation Standards: A guide for evaluators and evaluation users (3rd ed). Thousand Oaks, California: Sage.
Patton, M. (2010). Developmental Evaluation: Applying complexity concepts to enhance innovation and use. New York, NY: Guilford Press.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Thanks Judy, that’s helpful – regards – David
Thanks David, Janet, Chris and Jessica!
Seven semistructured interview questions used in the evaluation invited responses from stakeholders related to the purpose and motivation for the program; program goals, activities, experiences and resources; short and long term consequences; and also program design and implementation.
Analysis of stakeholder responses revealed finding situated within six themes. This example is from the theme of communication.
1. Findings Statement: University and community stakeholders identified different challenges and varying needs for communication related to the implementation and operation of the XXX program.
2. The Question: Which lines of communication are critical to the success of this XXX program?
3. Description, Extension, Connection: Several stakeholders reported the strength of the relationship between the university and the community in which they partner underpins the success of all program goals. Each stakeholder group acknowledged challenges in communication and expressed different information needs. All stakeholders acknowledged the importance of communication relating to duties, deliverables, and accountabilities.
Hi Judy, this does sound like a very interesting method. I would also like to know if you could provide any examples of the statements and the resultant questions. Thanks!
I too would really like to see an example that would further describe…
Step 1. Create the findings statement.
Step 2. Transform the statement into a question.
Step 3. Use stakeholder responses to describe, to extend understanding, or to make connections.
Hi Judy,
This method sounds very interesting.
Thank-you.
Thanks Judy, that’s an interesting contribution, helping to operatuionalise an aspect of the developmental approach.
I would find it helpful if you could provide some concrete examples from your experience of 1) the initial evaluation questions and 2) the eval findings turned into developmental questions.
With thanks, and best wishes – David McDonald