Hi, my name is Rick Axelson. I am a faculty consultant in the Office of Consultation and Research in Medical Education (OCRME) at the University of Iowa. OCRME works with faculty, staff, and students to support innovation and improvement in medical education practices. My colleague, Susan Lenoch, and I conduct workshops and individual consultations with clients – primarily educators – on program evaluation. The workshops attract faculty and staff from across the College of Medicine, all with different needs and interests.

One of the biggest challenges we’ve encountered in designing workshops for audiences of non-evaluators is “where to start.” Participants’ background knowledge seems to vary greatly and, there isn’t sufficient interest, time, or commonality in background on which to base our discussion. So, where do we begin?

Hot Tip: Start at the end. That is, build backwards from the participants’ common goals or interests. Why are they interested in doing an evaluation? Is there a particular product that they need to develop? To provide a common interest for participants, we advertise our evaluation workshops as an aid for completing the evaluation portion of an in-house grant application. Participants are encouraged to bring their grant proposal ideas to the workshop; we open the workshop by discussing their project ideas and refer to their projects throughout the workshop to illustrate evaluation principles. The added advantage of this approach is that it also works as a warm-up activity.

A second major challenge with these workshops is to provide a simple, straightforward process that will enable participants to create their desired evaluation product. Here, we have found the following to be helpful.

Hot Tip: Use examples of completed products to illustrate the evaluation design process. We walk participants through an example of a previously funded project. Breaking the evaluation design process down into a few simple steps, we show how the sample evaluation would have been developed using this method. We also point out areas (e.g., statistical analysis) where participants may need expert assistance to complete their study design.

Rad Resources: We have developed a workbook to lead participants through the development of the evaluation section for an in-house grant application. A previously funded project was used as the central example throughout the workbook to illustrate the steps involved in designing a program evaluation for an educational intervention. You can download it from: http://www.healthcare.uiowa.edu/ocrme/teach_train_sup/online_lean_teach_edu.htm

This aea365 Tip-a-Day contribution comes from the American Evaluation Association. If you want to learn more from Rick on this topic, check out the sessions sponsored by the Teaching of Evaluation TIG on the program for Evaluation 2010, November 10-13 in San Antonio.

grants, teaching

Greetings colleagues. I’m Kathleen Kelsey, a professor of Agricultural Education, Communications and Leadership at Oklahoma State University. I teach a variety of graduate courses including Program Evaluation. My students come from the agricultural sector and will work as agricultural teachers, Cooperative Extension educators, and in agricultural sales and services. They most likely won’t become professional evaluators. This is the only evaluation course they will take, so I strive to teach the theory and logic of evaluation from an applied perspective. I use a variety of techniques to motivate my students and peak their interest in program evaluation such as using film to teach ethics and evaluative thinking.

Hot Tip: I use the film Capote (2005) staring Philip Seymour Hoffman to teach about AEA’s Guiding Principles as an example of how an author uses any means necessary to get his story, including violating nearly every ethical principle outlined in the Guiding Principles. How do I guide student through this exercise?

1. I ask the students to review the Guiding Principles for Evaluators specifically, C. Integrity/Honesty, D. Respect for People, and E. Responsibilities for General and Public Welfare.

2. After reviewing the Guiding Principles, we watch the film, Capote (2005) (ASIN: B000E33VWW).

3. Students discuss and write about the ethical dilemmas posed by Capote’s behavior in regard to:

a. Gaining access to the research site

b. Financing the research project

c. His relationships with his research subject, Perry and others

d. How his actions affected society at large (considering the contextual factors associated with his work, for example  the political and social climate and the implications of his actions)

e. His consideration of the risks, harms, and burdens that might befall those involved in the research

f. Maintaining a balance between the client’s needs and his needs

4. At the conclusion of the film, Capote notes that his life was “changed forever” as a result of writing the book, In Cold Blood. I ask students to hypothesize why they think his research project changed Capote’s life forever, stimulating critical thinking about the many implications of our work as evaluators, such as our use of power, our position in the research site, and balancing these roles with conducting high quality evaluation.

This exercise has made a lasting impression on my students and has been one of best uses of film in teaching evaluative concepts. I hope your students enjoy it as much as mine!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Capote, ethics, film, teaching

Hi, my name is Ellen Taylor-Powell; I am an evaluation specialist at the University of Wisconsin-Extension.  Evaluation capacity building is my passion. Over my career, I’ve had the pleasure to partner with extension educators who excel in making learning relevant, practical and fun!  I’d like to let you know about two resources that bring together some of the teaching/learning materials we’ve developed and use in our own community capacity building work here in Wisconsin.

HOT TIP: When working with community-based organizations and practitioners, making evaluation practical and engaging is the name of the game!  One way to do this is to use the principles of adult learning that start with where people are, respect and build on existing knowledge and experience, ensure relevance,  and use a mix of interactive methods and facilitation techniques with opportunity for hands-on learning and lots of practice.  We’ve incorporated these principles in developing activities for helping people engage in evaluation.

RAD RESOURCE: Building capacity in evaluating outcomes: A teaching and facilitating resource for community-based programs and organizations provides 93 activities and materials (handouts and powerpoints) in eight units that cover the core topics of evaluation: getting ready; planning; engaging stakeholders; focusing the evaluation; collecting data; analyzing data; using data; and managing an evaluation. There is content guide and a facilitator’s guide with ideas of different educational methods to use from affinity diagramming to carousel brainstorming to buzz sessions; how to use  energizers and learning peripherals to enhance learning and lots of ice breakers and other facilitation tips.  You can download the pdf file and powerpoint files or order the full binder.
http://www.uwex.edu/ces/pdande/evaluation/bceo/index.html

RAD RESOURCE 2:  Developing a logic model: Teaching and Training Guide. This resource includes 17 activities with handouts and powerpoint slides to help community groups and program staff learn about and develop basic logic models.  There are sample workshop agendas, text for learning peripherals, ice breakers and for each activity, you will find its purpose, materials needed and process for conducting the activity.
http://www.uwex.edu/ces/pdande/evaluation/pdf/lmguidecomplete.pdf

This week’s posts are sponsored by AEA’s Extension Education Evaluation Topical Interest Group (http://comm.eval.org/EVAL/Extension_Education_Evaluation/Home/Default.aspx) as part of the EEE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting evaluations in an Extension Education context.

Capacity Building, extension education

Hello! My name is Sheila Kohn, and I teach Program Evaluation Methods at the University of Rochester’s Warner School of Education. I’d like to share a Cool Trick and Hot Tip for other evaluation educators.

I was very excited by AEA’s announcement of a daily evaluation blog and have faithfully read each entry since January 1. In fact, it has become part of my morning routine! I have my oatmeal, my coffee, and check out each day’s blog.

Cool Trick: I wanted my evaluation students to read and appreciate what the blog entries have to offer, so I created an assignment. Since one of the purposes of the course is for students to “develop the knowledge and ability to become savvy and critical users of evaluation information,” having them read the AEA blog is a perfect fit!

Here is the assignment: Read at least 3 entries from the aea365 Daily Tips Blog. Share with the class one Hot Tip, Cool Trick, or Rad Resource and how you might use it. Please share with the class the date and author of the blog entry so that others may readily find it if they wish. ***Note: No two people can share the same Hot Tip, Cool Trick, or Rad Resource, so be prepared with more than one!

Hot Tip: I passed around a sign-up sheet the first night of class so students could choose when in the semester they want to share. Of course, the earlier they do so, the more entries that have to choose from! And since 3 students share their chosen entry with the class each night, they have to communicate beforehand in order not to repeat an entry.

The blog entries shared by students serve as excellent catalysts for rich discussion about the nature, nuances, and complexity of evaluation work.

I’m finding that this assignment also engenders collegiality and a sense of community in our classroom, and I hope at least a few students become devoted blog readers like me and perhaps even future contributors!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

AEA resources, aea365, teaching

Hi!  We are Carrie M. Brown (student) & Angela Walmsley (professor) from Saint Louis University.  Today we will be sharing some tips on having a successful semester-long evaluation experience.

Graduate students can become discouraged to conduct a program evaluation by thinking it will take too long to complete and by not having direct assistance from a supervisor. We decided to tackle this obstacle by creating a one-on-one program evaluation course. Together, we completed a program evaluation within a semester time-frame that investigated an economics educational program for 3^rd and 4^th graders at a children’s museum. The evaluation implemented qualitative and quantitative methods, participant triangulation, and methods triangulation. Upon conclusion of the semester, the museum received a final written report. Here, we give you tips on how to create a successful semester-long one-on-one program evaluation course.

Hot Tips: Here is how we structured our course:

• Weekly meetings: One to two hours each, between professor and student.
• Evaluation journal articles: Student located one article per week and wrote a one-page summary/critique.
• Outlines of textbook chapters: Student created an outline, one chapter per week, from Program Evaluation: Methods and Case Studies (Posavac & Carey, 2002).
• Program evaluation: Weekly progress on the evaluation.
• E-mail phone contact as needed.

Hot Tips: Here are a few bits of advice on actually conducting the evaluation itself.

• Consider programs already in your institution or programs in the community that do not receive or cannot afford evaluations.
• Be prepared to communicate what you can provide and your time frame to the stakeholders.
• Contact and meet stakeholders ASAP, and set up a time to visit the program.e
• Make a site visit and learn the program ASAP.
• Avoid being too ambitious. Choose only one type of evaluation (needs, process, outcome, etc.).
• Determine a small set of goals for the evaluation.
• Create a timeline of weekly goals, keeping the end of the semester in mind.
• Choose a study design that fits the goals and timeline.
• Work on writing the report throughout the semester and work together to edit in sections.
• Review several reports to learn how to best format yours.

We found our experience to be both complete and satisfying. Perhaps you will consider a semester-long one-on-one program evaluation course!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

education, learning, teaching

Greetings colleagues. My moniker is Michael Quinn Patton and I do independent evaluation consulting under the name Utilization-Focused Evaluation, which just happens also to be the title of my main evaluation book, now in its 4^th edition. I am a former AEA president. One of the challenges I’ve faced over the years, as many of us do, is making evaluation user-friendly, especially for non-research clients, stakeholders, and audiences. One approach that has worked well for me is using children’s stories. When people come to a meeting to work with or hear from an external evaluator, they may expect to be bored or spoken down to or frightened, but they don’t expect to be read a children’s story. It can be a great ice-breaker to set the tone for interaction.

Hot Tip: I first opened an evaluation meeting with a children’s story when facilitating a stakeholder involvement session with parents and staff for an early childhood/family education program evaluation. The trick is finding the right story for the group you’re working with and the issues that will need to be dealt with in the evaluation.

Rad Resource: Dr. Seuss stories are especially effective. The four short stories in Sneeches and Other Stories are brief and loaded with evaluation metaphors. “What was I scared of?” is about facing something alien and strange — like evaluation, or an EVALUATOR. “Too Many Daves” is about what happens when you don’t make distinctions and explains why we need to distinguish different types of evaluation. “Zaks” is about what happens when people get stuck in their own perspective and can’t see other points of view or negotiate differences. “Sneeches” is about hierarchies and status, and can be used to open up discussions of cultural, gender, ethic, and other stakeholder differences. I use it to tell the story, metaphorically, of the history of the qualitative-quantitative debate.

Hot Tip: Children’s stories are also great training and classroom materials to open up issues, ground those issues in a larger societal and cultural context, and stimulate creativity. Any children’s fairy tale has evaluation messages and implications.

Rad Resource: In the AEA eLibrary I’ve posted a poetic parody entitled “The Snow White Evaluation,” that opens a book I did years ago (1982) entitled Practical Evaluation (Sage, pp. 11-13.) Download it here http://ow.ly/1BgHk.

Hot Tip: What we do as evaluators can be hard to explain. International evaluator Roger Mirada has written a children’s book in which a father and his daughter interact around what an evaluator does. Eva is distressed because she has trouble on career day at school describing what her dad, an evaluator, does. It’s beautifully illustrated and creatively written. I now give a copy to all my clients and it opens up wonderful and fun dialogue about what evaluation is and what evaluators do.

Rad Resource: Eva the Evaluator by Roger Miranda. http://evatheevaluator.com/

communication, Eva, metaphors, stories

Hi, I’m Bonnie Stabile, an Adjunct Professor in the School of Public Policy, and the Department of Public and International Affairs at George Mason University, just outside of Washington, DC.  I have been teaching Program Evaluation in the Master’s in Public Policy, and the Master’s in Public Administration, Programs for three years now.  This year I instituted a “Student Practitioner Speaker Series” in which I have students who have taken the class in previous semesters come back to talk to my current students about how they have applied the evaluation principles and techniques they have learned in the class on the job.

Hot Tip: Offer Master’s students the opportunity to adapt the semester research paper to focus an evaluative eye on a project of importance to them at work.  While most students, who range in experience from recent college graduates to established career professionals, opt for the standard research paper option of doing a GAO-style evaluation synthesis of a federal program, a few jump at the chance to work on a project of practical importance to them, while applying the theories and skills of evaluation.  I have had one student, director of a local county library system, undertake a needs assessment for library services, for which there was a keen felt need in light of threatened budget cuts.  Another student, neighborhood coordinator for a local county department of public works, developed a needs assessment to determine the training requirements for property code inspectors after a work reorganization.  These students approached their task with unprecedented energy and enthusiasm, and have expressed that the projects they undertook, beginning with their evaluation class papers, actually had lasting and demonstrable utility in the workplace.  As an added bonus, they have come back to share their success stories with current students, who benefit from hearing of evaluation success stories accomplished by their peers.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

teaching

My name is Trina Willard and I currently exercise my evaluation and measurement skills as the Vice President of Transformation Systems, Inc., a management consulting firm. During my 15 years in the evaluation field, I have had wonderful opportunities to engage across all levels of service organizations, including work with executives, leadership teams and service delivery staff.

I have personally gained insight about successful evaluation strategies by working “in the trenches”, and a great appreciation for the challenges that service providers face daily in juggling multiple priorities. As a frequent consultant to nonprofit and government groups, I consistently find that these organizations are most successful when armed with a foundational understanding of evaluation. However, competing demands, particularly in relatively small organizations, can preclude attention to professional development on the evaluation front. In fact, sometimes evaluation is tackled as an afterthought detrimentally, and only considered after all other priorities are addressed. Consequently, I believe that building evaluation capacity in the nonprofit sector often “sticks” when it is presented as a process of incremental steps, created systematically over time. In addition, such organizations are often most receptive to a practical, applied approach to evaluation, as opposed to a predominantly academic perspective. I’d like to recommend a rad resource that nicely taps into both of these needs.

Rad Resource: Hallie Preskill & Darlene Russ-Eft (2005). Building Evaluation Capacity: 72 Activities for Teaching and Training. Thousand Oaks, CA: Sage Publications. http://www.amazon.com/Building-Evaluation-Capacity-Activities-Teaching/dp/0761928103

Preskill & Russ-Eft do a great job of translating evaluation models, approaches, and techniques into relatable, hands-on exercises. The book is actually light on narrative “explanation”, but rather creates understanding through the direct implementation of tools and templates. As a trainer, I’ve used this resource repeatedly to illustrate evaluation principles for decision-makers and staff. One of my favorites: Activity 3, Evaluating Chocolate Chip Cookies Using Evaluation Logic. This exercise is always a hit at training events – a true example of learning in an enjoyable way! In addition, the text covers a wide variety of evaluation-relevant content, spanning topics such as ethics, political context, logic models, data collection, qualitative and quantitative analysis, budgeting for evaluation, and organizational buy-in. The layout easily facilitates training on one focused topic, or alternatively creation of a comprehensive training program.

I encourage you to give this resource a look. I’ll be interested to hear what you think!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Capacity Building, nonprofit, teaching, training

My name is John LaVelle, I am an advanced graduate student at Claremont Graduate University.  When I worked as the Jobs Coordinator for my department, I would encourage the students to develop a personal statement about evaluation.  This is important because when they would go to interviews, they would often be asked to describe their understanding of evaluation and explain it to people that may or may not have an background in evaluation.  This exercise eventually became an important element in the Evaluation Procedures course.

Hot Tip: Develop a personal statement of what evaluation means to you and how it can and should be practiced in dynamic, fluid, and political organizational and community environments and how it differs from basic research. In other words, if a client asked you to explain your understanding of evaluation, your approach to evaluation, how you would work with stakeholders, and so on, what would you tell him or her? In your statement, explain what processes you think are important for designing and implementing an evaluation, and how you would approach determining an evaluation’s design and data collection methods.

What might your personal statement of evaluation look like?

interviews, Jobs, teaching

My name is Kylie Hutchinson.  I am an independent evaluation consultant and trainer with Community Solutions Planning & Evaluation.  I am also a regular facilitator of the Canadian Evaluation Society’s Essential Skills Series course.  One question my students regularly bring up when learning to develop logic models and evaluation plans is the issue of “contribution analysis”, i.e. contribution versus attribution of a program’s activities to long-term outcomes and impacts.

Rad resource: I’ve found it useful to show them the “Output Outcome Downstream Impact Blues” karaoke video.  This short musical clip, written and performed by Terry Smutylo, addresses this issue in a humorous, yet informative way.  Although it was originally written for those working in international development, it is applicable to other evaluation sectors.  You can find Terry’s clip at http://www.idrc.ca/en/ev-65284-201-1-DO_TOPIC.html

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

contribution analysis, logic models