Hi, my name is Susan Sloan. I’m a program evaluator with about 20 years of experience—first as an evaluation team leader for Duerr Evaluation Resources in California and now as in internal program evaluator for Whatcom County Health Department (WCHD) in the Pacific Northwest.

As a small local health department, we are always looking for ways to increase internal evaluation capacity without adding additional staff. If you’ve been reading the news lately, you’ll know that local public health resources are decreasing at an alarming rate. This makes it even more important that the programs we run are effective and that our remaining staff is trained to understand evaluation and to participate as part of an evaluation team when needed.

In order to improve organizational evaluation capacity, I’ve used the Centers for Disease Control and Prevention’s Framework for Program Evaluation in Public Health as a teaching tool (MMWR 1999; 48 (No. RR-11): http://www.cdc.gov/eval/framework.htm

When I first discovered CDC’s Framework, I was amazed at how well it mimicked the evaluation process I had used for years to evaluate school intervention programs. The best feature of the framework is that it is an easy-to-understand, easy-to-teach six-step process for evaluation. Here at WCHD, we used our Community Health staff meetings to teach the framework over a six-month period. In order to make the training come alive, we used examples from an in process evaluation of our Children with Special Health Care Needs (CSHCN) program along with a staff-created evaluation of a mythical public health trails infrastructure campaign. The culminating activity resulted in a short report that was written by staff.

Hot Tip: The first AEA Coffee Break focused on DoView®, a modeling software developed by Dr. Paul Duignan: http://www.doview.com/ We have purchased several copies of this software as a wonderful augment to our use of the CDC Six-Step Framework. We are now able to create evaluation models that work us through the framework. Our DoView® models include: (1) a program overview, including  overall goals and major program components, (2) a comprehensive listing of all internal and external stakeholders, (3) a flow chart of each major program component, (4) a logic model, (5) an evaluation design (including the evaluation mission, major questions, methods, assignments, and timelines), and (6) reporting of evaluation findings. All of this can be easily shared with team members or partners either through DoView files, pdf’s, or HTML documents.

The CDC Framework combined with the DoView® software has allowed us to create an evaluation toolkit that meets the ever-challenging needs for local health department evaluation capacity building.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

CDC, evaluation capacity, logic models, Public Health

My name is Gerri Spilka and I am the Executive Director of the OMG Center for Collaborative Learning – a consulting firm that assists and assesses change through evaluation, strategic research and planning, grants management and capacity building. Among our many projects, OMG serves as the lead management team on the Robert Wood Johnson Foundation Evaluation Fellowship Program and it is about that program that I will share information today.

The skill set needed to interpret effectively public health, social service, community and organizational social patterns depends on the authentic and diverse experiences within program evaluation teams. Programs and services that are often evaluated traditionally work with underserved and disadvantaged communities, which are themselves becoming more diverse. However, the program evaluation field is becoming less diverse due to a lack of training and placement opportunities for emerging professionals from graduate programs who come from underserved and disadvantaged communities. This creates a problem for the field where, more often than not, high-quality evaluations reflect a process that incorporates diverse perspectives. The RWJF Evaluation Fellowship Program works to ameliorate this disparity.

Resource: The RWJF Evaluation Fellowship Program seeks to extend the skills of participants from diverse backgrounds to include culturally responsive evaluation practices. The Fellows program supports two cohorts annually on parallel paths:

• The Emerging Professionals Program is aimed at those who have graduated from a master’s or doctoral degree program within the last three years and provides a one-year placement, extensive training opportunities, and a support stipend
• The Retooling Professionals Program is focused on mid- or senior-level nonprofit professionals who remain in their current position, but receive additional training to extend their skills to evaluation. Their organization receives a $5,000 financial award and the professional receives a travel stipend.

Both cohorts receive customized training, facilitated by some of today’s thought leaders in field, as well as mentorship, and practical field-based guidance.

Hot Tip: While the deadline from participating in the Retooling Professionals program has passed, the deadline for applying for the Emerging Professionals program is coming up on July 31, 2010. The Emerging Professionals application may be downloaded online from http://www.rwjf-evaluationfellows.org/emerging-fellows.

Resource: The RWJF Evaluation Fellowship Program just released a video entitled Evaluation With a Diversity Lens. This extended discourse among a range of professionals explores the value of diversity in the evaluation profession and may be downloaded for free.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

culturally responsive evaluation, diversity

Hi, my name is Ellen Taylor-Powell; I am an evaluation specialist at the University of Wisconsin-Extension.  Evaluation capacity building is my passion. Over my career, I’ve had the pleasure to partner with extension educators who excel in making learning relevant, practical and fun!  I’d like to let you know about two resources that bring together some of the teaching/learning materials we’ve developed and use in our own community capacity building work here in Wisconsin.

HOT TIP: When working with community-based organizations and practitioners, making evaluation practical and engaging is the name of the game!  One way to do this is to use the principles of adult learning that start with where people are, respect and build on existing knowledge and experience, ensure relevance,  and use a mix of interactive methods and facilitation techniques with opportunity for hands-on learning and lots of practice.  We’ve incorporated these principles in developing activities for helping people engage in evaluation.

RAD RESOURCE: Building capacity in evaluating outcomes: A teaching and facilitating resource for community-based programs and organizations provides 93 activities and materials (handouts and powerpoints) in eight units that cover the core topics of evaluation: getting ready; planning; engaging stakeholders; focusing the evaluation; collecting data; analyzing data; using data; and managing an evaluation. There is content guide and a facilitator’s guide with ideas of different educational methods to use from affinity diagramming to carousel brainstorming to buzz sessions; how to use  energizers and learning peripherals to enhance learning and lots of ice breakers and other facilitation tips.  You can download the pdf file and powerpoint files or order the full binder.
http://www.uwex.edu/ces/pdande/evaluation/bceo/index.html

RAD RESOURCE 2:  Developing a logic model: Teaching and Training Guide. This resource includes 17 activities with handouts and powerpoint slides to help community groups and program staff learn about and develop basic logic models.  There are sample workshop agendas, text for learning peripherals, ice breakers and for each activity, you will find its purpose, materials needed and process for conducting the activity.
http://www.uwex.edu/ces/pdande/evaluation/pdf/lmguidecomplete.pdf

This week’s posts are sponsored by AEA’s Extension Education Evaluation Topical Interest Group (http://comm.eval.org/EVAL/Extension_Education_Evaluation/Home/Default.aspx) as part of the EEE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting evaluations in an Extension Education context.

Capacity Building, extension education

My name is Boris Volkov, and I am Assistant Professor in Evaluation Studies at the University of North Dakota. I’m involved with internal evaluation (IE) and evaluation capacity building (ECB) so I’ll share couple relevant tips and resources.

We know that the ECB practice is different from program evaluation. Obviously, the goal of ECB is to reinforce and sustain effective evaluation practices. However, properly structured internal evaluation practices can boost evaluation capacity development.

Hot Tips: Research and practice showed that IE can be very useful for ECB when internal evaluators are willing and able to systematically:

• engage leadership in evaluation
• demonstrate value and use of evaluation
• seek to integrate evaluation into organizational culture
• develop organizational evaluation strategy
• expand the social network of evaluation champions
• promote staff development in evaluation
• engage staff in evaluation activities (learning by doing)
• share their experiences and resources with staff, and
• build a meaningful organizational M&E database.

Rad Resource: If you think you are interested in doing at least some of the above, I would recommend reading about and using helpful strategies described in “Exploring effective strategies for facilitating evaluation capacity development” by Hallie Preskill and Shanelle Boyle (in a new UNICEF-sponsored book “From policies to results: Developing national capacities for country M&E systems,” available for free download at http://www.mymande.org/?q=virtual). Just to highlight some examples of how some of those strategies can be applied by internal (and, sometimes, external) evaluators:

• Involve your clients in your evaluation processes (whenever it is feasible/meaningful).
• Coach/mentor one or two colleagues in your organization by providing individualized evaluation/research support.
• Provide targeted trainings to your colleagues (e.g., on logic models, qualitative analysis, survey development).
• Produce, adapt, and share written resources about evaluation processes and findings pertinent to your organization (plans, reports, newsletters, memos, etc.).
• Tap the power of technology to learn from and about evaluation using the organizational website, Intranet, listserv and creating a database of internal and external evaluation documents/resources (forms, checklists, audio/video recordings, as well as written documents mentioned above).
• Promote communities of practice (or learning circles) that focus on or incorporate elements of evaluation – to share evaluation experiences and information among staff members and partners with common professional interests.
• Organize regular meetings to discuss evaluation related issues (e.g., as part of staff /R&D meetings).

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Capacity Building, ECB

My name is Beverly Parsons and I’m the executive director of InSites,  a non-profit research, evaluation, and planning organization. We use a systems orientation and conduct evaluations related to education, social services, community change, and health. I’m an AEA board member. I have a tip about how to build evaluation capacity through a type of Community of Practice.

Hot Tip: Consider using Communities of Learning, Inquiry, and Practice (CLIPs) to build evaluation capacity and develop a culture of inquiry across an organization.

CLIPs (a type of Community of Practice) are informal, dynamic groups of organizational members who learn together about their professional practice. They gather and analyze data about a question of importance to them. CLIP members learn an evaluative inquiry process with three basic steps: (1) design the inquiry; (2) collect data; and (3) make meaning and shape practice. The process has some special features to create continual renewal in the organization.  At Bakersfield College where we developed this process under a National Science Foundation grant, the CLIP members are faculty and staff. They focus their inquiries on student learning and success.

Typically, each CLIP consists of three to seven people with one person as the group facilitator. An overall CLIP Guide supports the work of multiple CLIPs at the organizational level, builds strategic linkages among the CLIPs, and connects the whole process appropriately to the organization’s other processes and initiatives. CLIPs support, and are supported by, the broader organization’s goals. CLIPs are adaptable for use in a variety of settings.

Hot Tip: The following features of CLIPs are especially important:

• Within general parameters including a focus on the organization’s core mission, CLIPs have the freedom to select their own members and topics; set their schedules; determine their budget allocations; and tailor the inquiry process. This freedom builds internal motivation among participants and helps ensure use of results.

• The CLIPs simultaneously focus on collaboration and inquiry, building a synergy that motivates completion of their investigation.

• The CLIPs use guiding principles that create an energizing learning environment and promote a natural flow from inquiry to change in practice. The CLIP members are learning at all stages of the inquiry process and readying themselves for a natural shift in practice.

Rad Resources: An overview video and modules about the CLIP process are free through InSites at www.insites.org/clip. Also an article, Evaluative Inquiry in Complex Times, that addresses the link to complexity science is available at http://www.insites.org/clip/clip_reports.html .

Feel free to contact me if I can be of assistance (bparsons@insites.org). I love working with the CLIP process. Perhaps part of the reason is it’s the only time I got a standing ovation from faculty (CLIP members) for my work related to evaluation!

CLIPs, collaboration, communities of practice, learning communities

My name is Sally L. Bond, President & Senior Consultant, The Program Evaluation Group, LLC. In 2004, Marilyn Ray (Finger Lakes Law & Social Policies Center, Inc.) and I began development of a peer review process for the American Evaluation Association’s Independent Consulting TIG. One of the defining objectives of the IC TIG is to provide members, especially sole proprietors and those in very small firms who often “fly without a net,” with support mechanisms to enhance the quality of their evaluation work. Therefore, we conceptualized the IC TIG’s peer review process as a professional development opportunity for our members, both reviewers and reviewees. The peer-review process engages professional evaluation colleagues to serve as critical reviewers of other colleagues’ evaluation reports with the purpose of providing feedback to inform and improve their practice.

Rad Resources: Although participation in the IC TIG’s double-blind peer review process is currently available only to members of the IC TIG*, the peer review guidelines and framework are accessible to anyone who is interested in improving the quality of written evaluation reports. You can find them online in the public AEA eLibrary at http://bit.ly/ICReview. The IC TIG’s “Framework for Peer Reviewers, Evaluation Reports” addresses many of the issues and report elements covered in the “Evaluation Report Checklist” by Gary Miron of Western Michigan University; see: http://www.wmich.edu/evalctr/checklists/checklistmenu.htm. The checklist is a useful tool for designing reports and collaborating with colleagues and clients on report preparation. The IC TIG’s peer review framework was developed for a different purpose. It was specifically designed to elicit detailed written feedback from professional peers who have technical expertise in program evaluation.

Hot Tip: Are you interested in doing some self-guided professional development in report writing? Use the “Framework for Peer Reviewers, Evaluation Reports” as a guide to reflect more deeply on the quality of your written work.

Hot Tip: Would you like to start a professional peer learning group in your evaluation office or company? Use the “Guidelines for Peer Reviewers” to orient colleagues to the purpose and expectations of the peer review process, then use the “Framework for Peer Reviewers, Evaluation Reports” to structure the group learning process.

Hot Tip: Are you part of a large organization that wants to initiate an internal double-blind peer review process? Use the “Instructions for Evaluators Submitting an Evaluation Report for Peer Review” and the “Submitting Evaluator Cover Sheet” to gather contextual information that is relevant to the review of the submitted report.

Hot Tip: Are you providing technical assistance to evaluation consumers who have little knowledge of what constitutes a really good evaluation report? Use the “Framework for Peer Reviewers, Evaluation Reports” to develop appropriate expectations for high quality deliverables.

*Both AEA and the IC TIG welcome new members. All members of the IC TIG are members of AEA. You can join the association – and subsequently the IC TIG – online at http://www.eval.org/membership.asp. Every AEA member may join up to five TIGs at no additional cost.

consultants, Evaluation, Peer Review

Hello, my name is Salvatore Alaimo and I am an Assistant Professor in the School of Public, Nonprofit and Health Administration at Grand Valley State University. I would like to share some tips on the evaluator’s role in evaluation capacity building with nonprofit organizations.

Evaluation Capacity Building (ECB) continues to gain momentum in the literature and in our profession thanks to scholars, researchers and practitioners such as Baizerman, Compton, & Stockdill; Bamberger, Rugh, & Mabry; Boyle, & Lemaire; Fetterman; Miller, Kobayashi, & Noble; Milstein, Chapel, Wetterhall, & Cotton; Patton; Presskill, & Russ-Eft; Sanders; Stufflebeam; Volkov, & King and others. Nonprofits have been challenged with meeting demands for evaluation from foundations, government agencies, the United Way and accrediting bodies, and face the question of what it takes to efficiently and effectively evaluate their programs.

These authors tell us that ECB is context dependent. The challenge we face as evaluators is determining what our specific role should be in ECB. Where is the line between helping a nonprofit organization develop evaluation capacity and becoming an enabler who contributes to co-dependency? Do we help the organization to continue without our assistance and work ourselves out of a job, or do we do just enough to get them started in the ECB process and leave them to continue to build capacity on their own? If we intervene too much, at what point are we taking on responsibilities and tasks best left for the organization’s stakeholders to build a culture for evaluation, mainstream it, and incorporate it into organizational learning?

These questions present challenges for our profession. There are tools we can use to help us navigate these dilemmas and incorporate into our decision making to strive to balance assisting nonprofits in ECB while leaving enough for them to enact on their own.

Hot Tip: I recommend two evaluation checklists by Stufflebeam and Volkov & King in the ECB category found on the Evaluation Center’s web site – http://www.wmich.edu/evalctr/checklists/checklistmenu.htm . I also recommend the program evaluation standards from the Joint Committee found on AEA’s web site at http://www.eval.org/EvaluationDocuments/progeval.html as well as the Guiding Principles for Evaluators at http://www.eval.org/Publications/aea06.GPBrochure.pdf . There are no magic pills or quick answers for working through the challenges of our role in ECB; however if you use these documents together in your ECB work, I believe you will find them extremely helpful in making wise choices and sound decisions.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Capacity Building, checklist, guiding principles, nonprofit

Hi my name is Melissa Rivera. I am the Director of Evaluation and Research at the National Center for Prevention and Research Solutions (NCPRS). Since 2006, our organization has collaborated with the National Guard Bureau to implement an evidence-based program, Stay on Track, to over 115,000 sixth through eighth grade students nationally. Since inception, NCPRS has strategically aligned our evaluation goals with the National Guard Counterdrug Program’s goals. Over the years we have learned several best practices that have helped us engage stakeholders throughout the evaluation cycle.

Hot Tip: A successful evaluation design requires engaging and empowering stakeholders throughout the evaluation cycle. We developed a comprehensive evaluation plan that reinforced and aligned:

• the goals and objectives of stakeholders
• the goals of national organizations
• programmatic objectives

To reinforce messaging, we ensured that training workshops and any materials developed consistently met identified goals.

Rad Resource: Rosalie Torres, Hallie Preskills, and Mary Piontek have created an effective tool that can be used to engage the stakeholders in evaluation strategies. In their book, Evaluation Strategies for Communicating and Reporting: Enhancing Learning in Organizations, they provide explicit details on how to develop a communication and reporting plan that can be used to engage stakeholders.

We have used these strategies and incorporated them into our evaluation plan and the results are promising. This information is also discussed during our training workshops and has provided reinforcement of our goals and objectives.

Hot Tip: Develop interactive multi-tier trainings that engage implementers and provide them with the tools they need to implement effectively and incorporate a certification element that enables them to train others. Some best practices include providing implementers with:

• the knowledge and resources they need to cascade the information appropriately.
• a procedural guide or a guidance document that contains an overview of the program, the design of the evaluation, how to administer surveys, how to return surveys, and a glossary of evaluation terms.

Collectively, engaging stakeholders equates to success in every facet of evaluation.

We hope that you consider using some of these tools!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

empowerment, engagement, stakeholders

My name is Susan Kistler,  and I am AEA’s Executive Director. I contribute each Saturday’s post to aea365.

I have been reading and thinking about learning from failure. For those of us who are data-lovers, who find security in information, it can be a challenge to overcome the tendency to want to collect all possible information, explore all feasible outcomes, before moving in a new direction. While I’ll never be one to dive in without testing the waters, I do want to go swimming a bit more often.

Rad Resource: Cannon and Edmondson’s 2004 paper Failing to Learn and Learning to Fail (Intelligently): How great organizations put failure to work to improve and innovate outlines three processes needed to fail intelligently – and positively:

1. Identifying Failure: “The key organizational barrier to identifying failure has mostly to do with overcoming the inaccessibility of data that would be necessary to identify failures.” (p. 10)
2. Analyzing and Discussing Failure: Create an open environment for discussing failures and overcome negative emotions associated with examining one’s own failures.
3. Experimentation: Put new ideas to the test, gather comparative data, and embrace both those that succeed and those that fail as contributing to the ultimate success of an endeavor.

Key takeaways include:

• “Most managers underestimate the power of both technical and social barriers to organizational failure.” (p.3) Technical barriers include ensuring stakeholders have the know-how to use and analyze data to learn; organizational barriers include rewarding only success.
• We must pay attention to and learn from small failures to prevent larger failures.
• “Creating an environment in which people have an incentive, or at least do not have a disincentive, to identify and reveal failures is the job of leadership.” (p. 13)
• “Conducting an analysis of failure requires a spirit of inquiry and openness, patience, and a tolerance of ambiguity. However, most people admire and are rewarded for decisiveness, efficiency and action rather than for deep reflection and painstaking analysis.” (p. 14)
• “It is not necessary to make all [stakeholders] experts in experimental methodology, it is more important to know when help is needed from experts with sophisticated skills.” (p. 26)

For me, Cannon and Edmondson reaffirmed the value of formal and informal evaluation and its role in innovation. They made it clear that data-lovers are uniquely positioned to fail intelligently.

Rad Resource: Want to think further about learning from failure? Try Giloth and Gewirz (from Annie E Casey) Philanthropy and Mistakes: An Untapped Resource in the Foundation Review (free online), or Bumenthal’s blog post on four steps to failing well (Fail Small, Fail Publicly, Fail to Win, Fail Proudly), or Pond’s Embracing Micro-failure (Sarajoy is an aea365 contributor, AEA member, and social entrepreneur!).

Note that the above reflects my own opinions and not necessarily that of my employer, AEA.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

failure, leadership, learning, organization