My name is Catherine (Brehm) Rain, Vice President of Rain & Brehm Consulting Group, Inc., an independent evaluation and consulting firm located in Rockledge, Florida.  I blog at The Evaluation Forum.

Rad Resource – The Evaluation Forum: New to our website, The Evaluation Forum focuses on the why and wherefore of evaluation of health promotion and health-related, risk-reduction programming.  The blog targets program personnel with some or no background in the principles, practices, purposes and benefits of program evaluation. Content is basic, and covers issues such as hiring an evaluator, program design, and fidelity (among other future topics). We post new content monthly and expect to increase frequency of postings this year.

Hot Tips – favorite posts: We added our blog in September of this 2011. Thus far, my two favorite posts are

• 10/3/2011 7 Qualities of an Effective Universal Program Design  Nine times out of ten, program ‘problems’ are directly related to program ‘design’—or the absence thereof. This blog covers basic qualities of program design and leads readers to one of my favorite places: Theory At a Glance: A Guide to Health Promotion Practice.
• 12/11/2011 Fiddling with Fidelity? Fidelity means, in a word: faithfulness.  As a former project director and a current evaluation team member specializing in Process Evaluation, I liken adherence to a grant management plan or a program design, to following a recipe for bread pudding.  Yes, you can tweak it here and there, if you know what you’re doing. If you don’t, you might end up as I did, with a batch of botched pudding!

Lessons Learned – why I blog: I blog, because I am first and foremost a writer—I write two other blogs non-related to evaluation.  Chiefly and with relevance to The Evaluation Forum: I blog to bring basic information to clients and program personnel so that they (a) grow their knowledge about evaluation; (b) apply evaluation principles to program design and implementation; and in so doing (c) maximize outcomes.

Lessons Learned: You have to commit to a blog in the same way you do to a subscribed newsletter: often; and whether you have time for it, or not. It is an adjustment.  It also takes time to develop a following—if you want one. Linking posts to our Facebook page has added a ‘friendly community’ factor, as well.  Sometimes, folks are a little shy of evaluation and its impact on their organization or project.  Finding us on-line or on Facebook with helpful hints or solid information they can use meaningfully, may be the first step we can take as professionals to help our clients and community succeed! (It’s also nice to be ‘liked’!)

This winter, we’re running a series highlighting evaluators who blog. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

blog · Evaluation

I’m Catherine Rain of Rain and Brehm Consulting Group, Inc., a research and evaluation firm in Rockledge, Florida. Ever look at an evaluation project overflowing with new learning and fantastic results and wonder: What now? What can we do with this valuable information –beyond writing a report or peer-review publishing?

How about taking it public with a communications strategy, targeting various segments of the community and the field?

Such campaigns:

• Educate policy-makers and decision-makers, as well as beneficiaries of services;
• Leverage data, bringing recognition and benefactors to the program or service;
• Positively re-frame attitudes about problems and risks affecting the community; and
• Demonstrate accountability and transparency among tax-supported providers of programs and services.

Okay: sounds like you might need to hire an expensive marketing firm, right?  Wrong! With a couple of no-cost tools, you can become the marketing pro!

Lesson Learned: Planning a communications strategy is similar to planning an evaluation.  Carrying it forward, beginning to end resembles the process of designing, implementing and evaluating a program or service.  Communicating with various ‘markets’ requires you to shape messages in the same way you tailor those directed at an evaluated population: with relevance, in a language they understand, and with sensitivity to their culture, values and traditions!

Rad Resource: The Pink Book, otherwise known by its longer name Making Health Communication Programs Work developed by the National Cancer Institute (NCI) at the National Institutes of Health (NIH) provides you with everything you need to know about planning, designing, implementing and evaluating a communication strategy. While the book addresses health communications, the strategies easily transfer to any evaluated discipline. Note: hard copies of the book are no longer available; however, you can download the document or print it in HTML and PDF versions (option to print page or whole document).

Lesson Learned:  As with evaluation and program implementation: Planning remains the key to success!

Rad Resource: Under contract, our firm authored the publication Translating Evaluation Results to Published Documents which summarizes Stage 1–Planning and Strategy Development—contained in the Pink Book. We analogize the approach to steps taken when designing and developing a program management or evaluation plan.

Hot Tip: There is a time and resource cost to producing an effective communications strategy.

Rad Resource: Funders often require evaluators to disseminate results and lessons learned. Ask if you can fund the communications strategy as a dissemination budget line item.

Hot Tip:  According to our colleague and AEA Executive Director Susan Kistler, evaluation roles are changing! We need to redefine our approaches with clients in meaningful ways. What better than to extend our communication avenues?

Rad Resource: Susan Kistler The Future of Evaluation: 5 Predictions (building on 10 others!) .

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Evaluation Communication

I’m Catherine (Brehm) Rain of Rain and Brehm Consulting Group, Inc., an independent research and evaluation firm in Rockledge, Florida. I specialize in Process Evaluation, which answers the questions Who, What, When, Where and How in support of the Outcome Evaluation. Field evaluations occur in chaotic environments where change is a constant. Documenting and managing change using process methods help inform and explain outcomes.

Lesson Learned: If you don’t know what or how events influenced a program, chances are you won’t be able to explain the reasons for its success or failure.

Lesson Learned: I’m a technology fan, but I’m also pretty old-school. Like Caine in the legendary TV show Kung Fu, I frequently conjure up the process evaluation ‘masters’ of the 1980s and ‘90s to strengthen the foundation of my practice and to regenerate those early ‘Grasshopper’ moments of my career.

Old-school? Or enticingly relevant? You decide, Grasshopper! I share a few with you.

Hot Tip:  Process evaluation ensures you answer questions of fidelity (to the grant, program and evaluation plan): did you do what you set out to with respect to needs, population, setting, intervention and delivery? When these questions are answered, a feedback loop is established so that necessary modifications to the program or the evaluation can be made along the way.

Rad Resource: Workbook for Designing a Process Evaluation, produced by the State of Georgia, contains hands-on tools and walk-through mechanics for creating a process evaluation. The strategies incorporate the research of several early masters, including three I routinely follow:  Freeman, Hawkins and Lipsey.

Hot Tip: Life is a journey—and so is a long-term evaluation. Stuff happens. However, it is often in the chaotic that we find the nugget of truth, the unknown need, or a new direction to better serve constituents. A well-documented process evaluation assists programs to ‘turn on a dime’, adapt to changing environments and issues, and maximize outcome potential.

Rad Resource:  Principles and Tools for Evaluating Community-Based Prevention and Health Promotion Programs by Robert Goodman includes content on the FORECAST Model designed by two of my favorites (Goodman & Wandersman), which enables users to plot anticipated activities against resultant deviations or modifications in program and evaluation.

Hot Tip:  If you short shrift process evaluation, you may end up with Type III error primarily because the program you evaluated is not the program you thought you evaluated!

Rad Resource: Process Evaluation for Public Health Research and Evaluations: An Overview by Linnan and Steckler discusses Type III error avoidance as a function of process evaluation. As well, the authors discuss the historical evolution of process evaluation by several masters including but not limited to Cook, Glanz and Pirie.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

community-based · process evaluation · Public Health