My name is Monica Hargraves and I work with Cooperative Extension associations across New York State as part of an evaluation capacity building effort in the Cornell Office for Research on Evaluation (CORE).  My work with Extension is shaped, in part, by insights we gained through a Concept Mapping research project we did in late 2008.  We wanted to explore, from practitioners’ perspectives, what factors contribute to supporting evaluation practice in an organization.

We used Concept Mapping software from Concept Systems, Inc. to gather ideas in response to this prompt: “One specific thing an Extension organization can do to support the practice of evaluation is …” Contributors included county-based educators and Executive Directors, as well as state-level Extension administrators and Cornell staff.  The raw ideas were pared down to a working set of 80, and then participants sorted the ideas into clusters and rated them on two criteria: potential for making a difference, and relative difficulty.

The individual ideas become points on a “Cluster Map” that gives a visual representation of how participants conceptualized the patterns and themes in ideas (see below. For information on the Concept Systems technology and the statistical techniques that underlie it, see www.conceptsystems.com.)  The ratings are useful for thinking strategically about what to do give priority to when trying to improve and sustain evaluation practice in organizations.

Rad Resource: For more detail on the study, including a handout with the individual idea statements and their ratings on potential difference, see http://core.human.cornell.edu/AEA_Conference.cfm#2008

Cluster Map of Ideas in Response to the Prompt: “One specific thing an Extension organization can do to support the practice of evaluation is …”

Lessons Learned:

• Technical assistance and training are not enough! The top-rated cluster in terms of potential for making a difference was “Communicate the Value of Evaluation.”  The ideas there included educating organization leaders, staff, and volunteers on the importance of evaluation (not the how-to), using evaluation results well and demonstrating how they lead to better programming, having an evaluation champion in-house, making evaluation results easy to understand and user-friendly.
• Communication is important. Communication should be used to motivate evaluation and build organizational commitment to it, and as a practical tool for sharing what works, fostering collaborations, and saving time.
• Leadership and Structure matter. The second and third most important clusters were “Set Expectations and Requirements” and “Integrate into Organization Structure”.  Respondents wanted clarity and consistency, and to have evaluation woven into a wide range of organization functions and practices.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

cluster map, data visualization

Hi, my name is Samantha Grant, and I work for the University of Minnesota Extension as an Extension Educator for Program Evaluation. Some of my work entails designing and implementing evaluations with youth in the 4-H program. I would like to share some tips when conducting evaluations with youth.

Hot Tip – Build relationships: “Kids don’t care what you know, until they know that you care.” This aphorism certainly helps to guide the way that youth workers interact with youth, but it should also influence evaluators. You shouldn’t expect to walk into a room of youth and have them give you honest feedback on an evaluation. Slow down. Take some time to get to know the participants, even if it means using more of your time. If you aren’t able to invest the time or think it influences the evaluation design, work with the leader- whether a teacher or youth worker- to administer the evaluation. If youth see that someone they trust is comfortable with the evaluation, they in turn will be more comfortable.

Hot Tip – Read: Don’t assume that a youth audience can read and interpret evaluations. Especially when working with an elementary audience, read the survey aloud. This helps to break down barriers, assuring that everyone is processing the information in a similar fashion. If you have an additional person helping with the evaluation, have them float throughout the room and check in with youth that are speeding ahead or lagging behind.

Hot Tip – Explain, explain, explain: As simplistic as it sounds, make sure to tell youth that an evaluation is not a test, and it will not be graded or influence their participation in the program. We are currently working with a youth climate that is saturated with standardized testing. It goes without saying that these experiences color perceptions of any form of assessment.

Also, as you administer an evaluation, make sure to answer clarifying questions. With a research study, you may not be able to give this clarification, but evaluation is usually built on the premise of getting the most useful feedback from participants. If youth don’t understand a word or what a sentence means, make sure to clarify. Your data will be much richer.

Hot Tip – Have fun: Youth have great insights. Conducting evaluations with youth will make you a smarter, savvier evaluator, so enjoy!

This week’s posts are sponsored by AEA’s Extension Education Evaluation Topical Interest Group (http://comm.eval.org/EVAL/Extension_Education_Evaluation/Home/Default.aspx) as part of the EEE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting evaluations in an Extension Education context.

extension education

Hi. My name is Heather H. Boyd and I am a program evaluator currently at Virginia Tech. Much of my work is with faculty and staff who are interested in learning about evaluation and improving their evaluation skills. My Ph.D. is in Mass Communications and I have been a practicing evaluator for more than fifteen years.

Lessons Learned: When I align evaluation techniques and concepts to faculty member goals for their educational and research programs, they are much more open to learning about evaluation and incorporating higher quality evaluation practice into their work. Few of the faculty and staff that I work with have formal training in evaluation, and much of their experience with evaluation has been limited to classroom-based feedback forms. So, I try to demonstrate easy ways for them to incorporate gathering evaluation evidence as part of their program routines. And, I show them how to formalize and systemize gathering credible evidence that they previously had not recognized as useful evaluation information. Because my role is evaluation capacity building in my public educational agency, taking small steps and completing small pilot projects fits within a developmental approach to building better evaluation skills and higher quality evaluation products.

This week’s posts are sponsored by AEA’s Extension Education Evaluation Topical Interest Group (http://comm.eval.org/EVAL/Extension_Education_Evaluation/Home/Default.aspx) as part of the EEE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting evaluations in an Extension Education context.

extension education

Hi!  My name is Lisa Townson.  I am Assistant Director of Programs at University of New Hampshire Cooperative Extension and 2010 Chair of the Extension Education Evaluation TIG. In this day in age, accountability and impact has become the mantra for many publically funded organizations.  This posting suggests ways to consider your target audience when collecting and reporting impact evaluation data.

The Cooperative Extension System (CES), a land-grant university-based outreach and educational organization, exists nationally in every state and territory of the United States with programs in 4-H youth development, agriculture, family development and community development.  The main driver for program evaluation in Cooperative Extension is public accountability to maintain and increase funding. With its reliance on multiple funding streams from local, state, federal, and nongovernmental sources, Extension strives to tell the story of program impact and public value to a variety of audiences.

I’m working with several of our local offices in New Hampshire and trying to help them do a better job of communicating the impact their programs make and it’s sometimes a challenge to get them to think about their impacts in a new way.  One key I’ve found is to ask them to try and look at their programs from someone else’s point of view.  This is very easy to suggest, but often more difficult to do.

Hot Tip: Who cares about my program and why?  If a legislator is considering funding for an organization such as Cooperative Extension, what will make them consider our programs necessary, not just nice to have?  Try to find out what kinds of interests your stakeholders have and tailor your evaluation and impact reporting to their interests.  If you collect data to communicate economic impact to an individual (or group of individuals) who are interested in environmental sustainability, then you’re not likely to be successful in your message.   On the other hand, if you know economic viability and job growth are important to your stakeholders, find a way to connect your programs to that.   Talk about how many jobs would potentially be created or saved or how farms you’re working with are able to increase profitability and in so doing, keep the local veterinarian and feed store in business.  If your audience wants to know about societal and growth impacts, report how many acres were put into conservation easements as a result of your program and remind them how many houses might have been built on that same acreage which might increase the need for services in a community such as emergency responders and teachers in the school.

Reporting impacts and tailoring an evaluation project to a stakeholder is important whether you are reporting to a foundation that funded your project or if you’re reporting to legislators that will determine what level of funding you will receive.  Know what’s important to the audience and find a way to collect the right impact data to tell a compelling story.

This week’s posts are sponsored by AEA’s Extension Education Evaluation Topical Interest Group (http://comm.eval.org/EVAL/Extension_Education_Evaluation/Home/Default.aspx) as part of the EEE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting evaluations in an Extension Education context.

extension education, tailoring

Hello, My name is Michael Duttweiler and I am assistant director for program development and accountability for Cornell University Cooperative Extension.  One of my roles is to provide evaluation planning assistance to local extension educators, many of whom do not have formal backgrounds in program development or evaluation.

Hot Tip: Focusing an Evaluation Using a Succinct Purpose Statement There are many resources that guide one through focusing an evaluation typically ranging through a series of considerations such as program life cycle, stakeholder interests, key questions to be answered, resource considerations, respondent access, evaluability, and the like.  Many people not steeped in evaluation practice find starting the planning process with such a broad array of considerations both complex and intimidating.  A number of years ago, I landed on the approach of starting first by asking people to draft a brief evaluation purpose statement. This is a one-paragraph description of their evaluation effort that describes what is and is not being evaluated and the purposes of the evaluation.  It sets boundaries by including a description of the program elements and time frame being considered, which audiences are being addressed, and which goals or objectives are of most interest.

Example: The purpose of this evaluation is to assess the effectiveness of the 2010 Master Forest Owners Workshop in supporting and prompting volunteers to extend their knowledge to other forest owners in their local communities.  A secondary purpose is to provide documentation and assessment information for use by persons considering replicating the model with other forest owner groups.  Considerations include program structure and processes, curricular choices, and short-term impact assessment.  Other means of supporting forest management volunteers such as our newsletter and quarterly conference calls will not be assessed.

The purpose statement becomes an anchor to which we return at each phase of the planning process modifying the statement as necessary and appropriate.  With a draft purpose statement in hand, I then move to identification of the one or two priority evaluation questions to be answered based on the purpose statement.  The usual timeframe, respondent access, and resource limitation questions can then be raised in context and with a clear sense of intent as expressed in the purpose statement and key questions.

This week’s posts are sponsored by AEA’s Extension Education Evaluation Topical Interest Group (http://comm.eval.org/EVAL/Extension_Education_Evaluation/Home/Default.aspx) as part of the EEE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting evaluations in an Extension Education context.

extension education, focusing

Hi, my name is Ellen Taylor-Powell; I am an evaluation specialist at the University of Wisconsin-Extension.  Evaluation capacity building is my passion. Over my career, I’ve had the pleasure to partner with extension educators who excel in making learning relevant, practical and fun!  I’d like to let you know about two resources that bring together some of the teaching/learning materials we’ve developed and use in our own community capacity building work here in Wisconsin.

HOT TIP: When working with community-based organizations and practitioners, making evaluation practical and engaging is the name of the game!  One way to do this is to use the principles of adult learning that start with where people are, respect and build on existing knowledge and experience, ensure relevance,  and use a mix of interactive methods and facilitation techniques with opportunity for hands-on learning and lots of practice.  We’ve incorporated these principles in developing activities for helping people engage in evaluation.

RAD RESOURCE: Building capacity in evaluating outcomes: A teaching and facilitating resource for community-based programs and organizations provides 93 activities and materials (handouts and powerpoints) in eight units that cover the core topics of evaluation: getting ready; planning; engaging stakeholders; focusing the evaluation; collecting data; analyzing data; using data; and managing an evaluation. There is content guide and a facilitator’s guide with ideas of different educational methods to use from affinity diagramming to carousel brainstorming to buzz sessions; how to use  energizers and learning peripherals to enhance learning and lots of ice breakers and other facilitation tips.  You can download the pdf file and powerpoint files or order the full binder.
http://www.uwex.edu/ces/pdande/evaluation/bceo/index.html

RAD RESOURCE 2:  Developing a logic model: Teaching and Training Guide. This resource includes 17 activities with handouts and powerpoint slides to help community groups and program staff learn about and develop basic logic models.  There are sample workshop agendas, text for learning peripherals, ice breakers and for each activity, you will find its purpose, materials needed and process for conducting the activity.
http://www.uwex.edu/ces/pdande/evaluation/pdf/lmguidecomplete.pdf

This week’s posts are sponsored by AEA’s Extension Education Evaluation Topical Interest Group (http://comm.eval.org/EVAL/Extension_Education_Evaluation/Home/Default.aspx) as part of the EEE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting evaluations in an Extension Education context.

Capacity Building, extension education

Hi!  My name is Nancy Franz, and I am a Professor and Extension Specialist at Virginia Polytechnic Institute and State University.  I’m also the past chair of the AEA Extension Education Evaluation TIG.  Today I’ll be sharing some information on different approaches used in extension education evaluation.

Hot Tip: As the figure below illustrates, community educators approach their work in a variety of ways. This model created by Dr. Merrill Ewert, a past director of Cornell Cooperative Extension focuses on two dimensions – process and content (Franz, 2002). Process refers to the methods used to deliver educational programs and content refers to the information shared that helps change learner’s knowledge or behavior. Combining these dimensions presents four domains that describe the ways extension work is conducted.

The first domain, “service,” includes functions conducted by educators that bring learners to the organization to help them solve a short term problem. Educators using this approach provide little process or content in delivering these services but find this work an important approach to building relationships with clientele groups and addressing basic clientele needs. Evaluation tends to focus simply on customer satisfaction with the service.

The second approach to community education, “facilitation,” finds educators serving as facilitators of group processes in educational settings rather than delivering content or information. Educators may facilitate public forums on community issues or strategic planning sessions for groups. Process evaluation is most appropriate for these efforts.

“Content transmission” as an educational approach focuses on the development and distribution of information including newsletters articles, web-sites, PowerPoint presentations and other resources. Evaluation with this approach most often focuses on knowledge gained from the information disseminated to learners.

The “transformative education” approach to education takes place when educators develop long-term educational relationships with clientele and focus on changes in learning and behavior that leads to change in social, environmental, and economic conditions (Franz, 2002). These efforts are intentionally designed to reach learners over time to address important issues rather than providing one-time educational events. Learners gain new competencies, apply what they’ve learned to their personal context, share results with each other, and adjust their application of learning as needed. This approach to community education most often focuses on impact evaluation to determine what changes have happened for learners, families, communities and organizations as a result of the educational program.

For more information, see Franz, N. (2002), Transformative learning in intraorganizational partnerships, Unpublished doctoral dissertation, Cornell University, Ithaca, NY.

This week’s posts are sponsored by AEA’s Extension Education Evaluation Topical Interest Group (http://comm.eval.org/EVAL/Extension_Education_Evaluation/Home/Default.aspx) as part of the EEE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting evaluations in an Extension Education context.

extension education