Hi, we are Christine Johnson and Terri Anderson, members of the Massachusetts Patient Centered Medical Home Initiative (MA-PCMHI). MA-PCMHI is Massachusetts’ state-wide, multi-site PCMH demonstration project engaging 46 primary care practices in organizational transformation to adopt the PCMH primary care model.  Our roles as Transformation and Quality Improvement Director (Christine) and Qualitative Evaluation Study Team Lead (Terri) require us to understand the 46 practices’ progress towards PCMH model adoption in distinct yet complementary ways.  Our colleagues sometimes assume that we must remain distant to conduct our best possible work.  Their concerns are that our close working relationship will somehow contaminate the initiative or weaken the evaluation’s credibility.  However, we find that maintaining our connection is vital for success of both of the initiative and the evaluation.  We’d like to share the following:

Lessons Learned:

• Transformation and Quality Improvement (Transformation/QI) and evaluation both seek to understand how the practices best adopt the PCMH model and to describe the practices’ progress.  To promote our mutual interest, we regularly attend each other’s team meetings. Doing so increases the opportunity to share our perspectives on the MA-PCMHI. To date the evaluators have advised some formative project adjustments while the MA-PCMHI intervention team has increased the evaluators’ understanding of the survey and performance data submitted from the practices. Currently, the project team and the evaluators collectively are establishing criteria to select six practices for in-depth site visits.
• Transformation/QI and evaluation often use the same data sources but in different ways.  Specifically, the practices use patient record data in their Plan-Do-Study-Act (PDSAs) cycles then submit the same data for the evaluation’s clinical impact measures.  The practices initially resisted this dual data use.  However, through our Transformation/QI-Evaluator connection we increased the practices’ understanding of how their use of data in the PDSAs improved their clinical performance which in turn improved the evaluation’s ability to report a clinical quality impact. Presently, performance data reporting for clinical impact measures and practices’ use of PDSAs have increased.

Hot Tip: Develop a handout describing the similarities and differences between research, evaluation and quality improvement.  Having this information readily available has helped us to address concerns about bias in the evaluation.

Rad Resources:

• To learn more about Transformation/QI see the Agency for Healthcare Research and Quality’s 2010 Consensus Report on Practice Facilitation for Primary Care improvement.
• To learn more about PDSAs see the Institute for Healthcare Improvement’s Plan-Do-Study-Act tools.

(Share Clip)

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Capacity Building · qualitative

I’m Kylie Hutchinson, AEA Member, independent evaluator and owner of Community Solutions Planning & Evaluation, occasional aea365 contributor, and annual writer of Christmas carols for evaluators and not-for-profits. Each year, I adapt a carol to share with my colleagues and clients. Last year, Susan Kistler asked if I would share one with aea365 and I’m back again for a second year.

Hot Tip: Find the humor in what you do!

Rad Resource:

The Internal Evaluator’s Carol
(sung to the tune of ‘O Come All Ye Faithful’)

Oh come all ye data,
For this pilot program,
Oh show me, oh show me,
Sig-nif-i-cance.
I have collected,
cleaned and analyzed,
Now prove that it’s effective,
Please prove that it’s effective,
Oh prove that it’s effective!
S-P-S-S.

I must remain,
Arms length and objective,
Although the management,
is pressuring me.
What will they say?
If I find no impact,
I must be strong, not waver,
I’m doing them a favor,
We’ll learn from this I wager,
M&E.

Time for statistics,
Chi Square and Regression,
T-test, ANOVA,
My fingers are crossed.
Thanks be to those for,
Program Eval Standards,
You are my point of reference,
But really it’s my preference,
I’m asking for with deference,
Sig-nif-i-cance!

Rad Resource: You can find other carols on my website here.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Christmas · humor

My name is Andrew Anderson and I’m the manager of an internal evaluation team in one of Australia’s largest charities, The Benevolent Society. One of our guiding principles is to make our evaluations useful. But when we surveyed our 800 staff two years ago they told us that, although they valued evaluation, they didn’t think the way our organisation delivers services had changed much as a result. This made us realise there’s an important role for internal evaluators in supporting and monitoring the implementation of evaluation recommendations. Here are some hot tips and lessons we’ve learned.

Lessons Learned:

• A culture of evaluation needs to be modelled at every level of the organisation.

• Evaluation recommendations will only be implemented if managers and service staff feel a sense of ownership over them and are involved in developing them. We learned, for example, that although it’s vital to have the support of the CEO and Senior Executives, middle management must also be brought into the process.

Hot Tips:

• Use a range of techniques to involve staff in the process and build their evaluation knowledge and skills at all points in the evaluation process, not just at the stage of generating recommendations. For example, we do this by including staff on evaluation reference groups, training staff in the collection and analysis of data, and involving staff in the development of evaluation questions and plans.

• Where possible, also involve staff in the analysis and interpretation of evaluation data. We do this by holding a one-day workshop with staff, and by the end of the day we have a set of agreed recommendations that are directly linked to the findings.

Rad Resource:

There is lots of literature on evaluation capacity building. We like Volkov and King’s evaluation checklist for building organisational evaluation capacity, and Preskill and Torres’ various publications on organisational evaluation capacity building. We particularly like Hallie Preskill’s book Building evaluation capacity: 72 activities for teaching and training. Her evaluating a chocolate chip cookie exercise is a great way of breaking the ‘evaluation ice.’

Lessons Learned:

We learned the hard way that having an organisational evaluation policy and related procedures is crucial to ensure accountability for implementing recommendations. An evaluation policy should be clear on roles and responsibilities of all staff involved, and include procedures for monitoring the implementation of recommendations.

Rad Resource:

There are some good examples of evaluation policies whose adoption may allow organizations to implement of evaluation recommendations. This brief paper by the UN outlines what an evaluation policy should contain and has links to examples.

The American Evaluation Association is celebrating Internal Evaluators TIG Week. The contributions all week come from IE members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

checklist · evaluation capacity building · roles

We are Hilary Loeb and Kelly Bay of the Research and Evaluation Department at the College Success Foundation.  Many of our scholarship and support programs host events in which we collect data from students and educators.  As internal evaluators, we often rely on colleagues to collect and enter survey data from these groups. The results are used for staff learning internally and external reporting.  To help evaluators increase survey relevance, decrease demands on respondents’ time, and ultimately boost data quality and response rates, below are tips on instrument design and data collection.

Lessons Learned:

Look for ways to make surveys easier for staff to administer up front and more useful to stakeholders at the back end. The key is keeping the main focus on your programs while building support for data collection and analysis efforts.

Hot Tips

Survey Design:

• Ensure that survey content is relevant: Meet with the entire program team and start with the question, “What do we want to learn about our program?” before discussing what’s needed for grant-reporting requirements.
• Draft a survey using previously tested questions:  You don’t have to reinvent the wheel. By using previously tested survey questions from existing “banks” of items, you can save time and often improve the quality of the data collected (see Rad Resources).
• Pilot test surveys with your program team and other stakeholders. This exercise never fails to elicit important feedback and takes only a modest amount of time. It’s amazing what fresh eyes can find! Where possible, use trainings and even Board meetings as opportunities to pilot and discuss surveys.

Survey Data Collection:

• Be strategic about paper versus online surveys: When event participants can’t readily access computers, paper surveys may help increase response rates.  Online surveys are more appropriate when participants are able and willing to access technology.
• Designate sufficient time and staff to collect survey data: Ensure that there is a specific time slot dedicated for survey completion. It should be near to but not at the very end of the event.  We suggest providing a script to help staff describe the survey’s purpose and value.
• Consider using scanning software for paper surveys: Scanning software automates data entry by reading the optical marks on paper survey forms, which can reduce errors and save time.  Before purchasing, it’s best to test. We piloted a Free Demo of Remark Office OMR, to confirm that this was the right software for our organization.

Rad Resources:

A Bing search of survey item banks yields over 60 million results.  Our favorites in the education and youth development field include:  Ansell-Casey Life Skills Assessments,  the Youth Behavioral Risk Surveillance System and  National Center for Educational Statistics resources .

The American Evaluation Association is celebrating Internal Evaluators TIG Week. The contributions all week come from IE members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

data collection tool · free tools · resources

My name is Alicia McCoy and I am the Research and Evaluation Manager at Family Life in Melbourne, Australia.  Family Life is a community service organization that offers a diverse range of services for families, children and young people, including family support, counseling, and mediation.

In 2009, Family Life established an internal research and evaluation unit. But building a culture that supports evaluation didn’t follow automatically. It’s taken time, patience, focus, and even a little humour.  It’s important to regularly communicate with staff about the value of research and evaluation.  Here are some of my tips and lessons learned for doing this well.

Lesson learned: It helps to have a channel 

The creation of an internal research and evaluation blog in early 2011 turned out to be one of the greatest contributors to the growing research and evaluation culture at Family Life.  Make your blog highly visible and easily accessible to staff.  Update it regularly to keep it fresh and interesting.

Lesson learned:  Think outside the square

Blogs can be used for much more than just posting and comments.  Ours includes current research and policy information, case studies, program evaluation summaries, an acknowledgements register, key resources, relevant links, and even competitions.  It is a one-stop-shop for anything related to research and evaluation.

Hot tip: Be interactive

All staff members are encouraged to interact with the blog by guest-blogging on topics of interest to them and their work, and by commenting on and discussing the contributions of others.  These entries have been some of the most popular and have motivated others to give it a go.  Have a support process in place for those who are less confident with their blogging skills.

Hot tip: Be creative to boost relevance

Our blog delivers research and evaluation to staff in informal and creative ways. Examples include using a media story to provide context for a piece about a current policy initiative, or using an anecdote to provide the ‘hook’ for an entry on a current research study.  Feedback from staff suggests that these kinds of approaches help them better understand research and evaluation and makes it more interesting. The upshot: they are more likely to use information from evaluations and research studies.

Hot tip: Evaluate your own progress

It can take several months or longer for a blog to become part of a culture, especially if your organization is new to blogging.  It is important to regularly measure the popularity and use of the blog to see how you’re tracking.  Successful blogs will gradually increase the number of monthly visits until the majority of staff are engaged.

The American Evaluation Association is celebrating Internal Evaluators TIG Week. The contributions all week come from IE members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

blog · Capacity Building

My name is Rahel Wasserfall. I am an internal evaluator and do program development for the International Summer School on Religion in the Public Life (ISSRPL). I’m also affiliated with the Women Studies Research Center at Brandeis University and use mostly qualitative methods, including participant observation and in-depth interviews.

As an evaluator with nine years at the ISSRPL, I’m steeped in my organization’s culture, comfortable with how our programs work and adept at recognizing familiar patterns and behaviors. While these are useful assets, to assess programs effectively, I also need ways to stay “apart.” A key question I’ve been asking is how can internal evaluators keep from falling into analytical ruts? How do we recognize the potential value of information that falls outside the norm or challenges our assumptions? In short, how do we see with “fresh eyes,” when needed?

Lessons Learned: Recognize outliers

• Those within a culture can usually tell what is normative and what falls outside the norm. I suggest paying close attention to the outlier story – information, cases, events and other occurrences that are atypical, when compared to the overall data collected.  Instead of dismissing such occurrences, I study them: they may signal a need to dig deeper for more insight.

• A nice example comes from a debriefing following a two-week international and inter-religious program on tolerance. One participant was surprisingly angry with the food: “Why didn’t we have pork?” While not the majority experience, the unexpected comment prompted my review and analysis of participants’ reactions to the food through the years. Ultimately, this yielded broader insights about how our program addressed minorities’ preferences and led to changes.

Hot Tips:

• When collecting qualitative data – for example, through interviews, focus groups, or participant observation – every occurrence, event, and piece of information is potentially useful. Take notes, even if it doesn’t seem to make sense right away.

• Pay special attention to strong emotions and opinions. These are often good clues to investigate further.

• Where possible, use follow-up discussions and debriefings with participants to better understand atypical behaviors and opinions. The added context can sometimes offer surprising insight, which may shift initial interpretations.

• Questioning our assumptions about data can be easier said than done. I find it helpful to do a follow-up review of events, behaviors, and opinions that I’ve decided not to include.  Asking why particular data won’t figure in the analysis can help make underlying assumptions explicit and can yield additional, valuable understanding.

Rad Resources:

• Livia Polanyi’s  “Stories as Cultural Texts,” in Telling the American Story.
• Gregory Bateson: Steps to an Ecology of the Mind  (PDF may load slowly)

The American Evaluation Association is celebrating Internal Evaluators TIG Week. The contributions all week come from IE members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

debrief · notes · outliers

My name is Stan Capela, and I am the VP for Quality Management and the Corporate Compliance Officer for HeartShare Human Services of New York. I have devoted my entire career to being an internal evaluator in the non-profit sector since 1978.

In graduate school, you develop a wide range of skills on how to conduct program evaluation. However, there is one skill that schools don’t focus on – how an internal evaluator develops a brand that clearly shows that s/he adds value to the organizational culture.

Developing a personal brand can be a challenge, given workplace perceptions, pressures, and stresses. For example, program staff may have varying perceptions of my dual roles as an internal evaluator, which involve supporting their efforts and pointing out deficiencies. In addition, I often conduct simultaneous projects that combine formative and summative evaluations and may involve quality and performance improvement. Finally, my attention often gets split between internal reports and external reviews.

Lesson Learned: Producing quality reports that clearly are utilization-focused is important. But I’ve found that the secret ingredient to making my work valued and developing a brand within the organization is simply the ability to help answer questions related to programmatic and organization problems.

Lesson Learned:  Get to know program staff and their work.  In my early years, I found it especially helpful to spend time talking to program staff. It provided an opportunity to understand their work and the various issues that can impact a program’s ability to meet the needs of the individuals and families served. Ultimately, this helped me to communicate more effectively with staff and about programs.

Lesson Learned:  Find additional outlets to build your networks. I have had an opportunity to be a Council on Accreditation (COA) Team Leader and Peer Reviewer and have developed contacts by participating in 70 site visits throughout the US, Canada, Germany, Guam and Japan. Over the span of 34 years, I have developed a network of contacts that  have helped me respond expeditiously – sometimes through one email – when a question arises from management. As a result, I became know as a person with ways to find answers to problems.

RAD Resources:   Many of my key resources are listservs.  These include Evaltalk – a listserv of worldwide program evaluators; the Appreciative Inquiry List Serve (AILIST); and the List of Catholic Charities Agencies (CCUSA).  Other helpful affiliations include the Council on Accreditation (COA), the Canadian Evaluation Society, and the American Society for Quality.

If you have any questions, let me know by emailing me or sharing them via the comments below.

The American Evaluation Association is celebrating Internal Evaluators TIG Week. The contributions all week come from IE members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

branding · formative · Networking · reports · summative · utilization-focused evaluation

Greetings aea365 community! I’m Ann Emery and I’ve been both an external evaluator and an internal evaluator. Today I’d like to share a few of the reasons why I absolutely love internal evaluation.

Lessons Learned: Internal evaluation is a great career option for fans of utilization-focused evaluation. It gives me opportunities to:

• Meet regularly with Chief Operating Officers and Executive Directors, so evaluation results get put into action after weekly staff meetings instead of after annual reports.
• Participate on strategic planning committees, where I can make sure that evaluation results get used for long-term planning.

Lessons Learned: Internal evaluators often have an intimate understanding of organizational history, which allows us to:

• Build an organizational culture of learning where staff is committed to making data-driven decisions.
• Create a casual, non-threatening atmosphere by simply walking down the hallway to chat face-to-face with our “clients.” I hold my best client meetings in the hallways and in the mailroom.
• Use our organizational knowledge to plan feasible evaluations that take into account inevitable staff turnover.
• Tailor dissemination formats to user preferences, like dashboards for one manager and oral presentations for another.
• Participate in annual retreats and weekly meetings. Data’s always on the agenda.

Lessons Learned: Internal evaluators can build evaluation capacity within their organizations in various ways:

• I’ve co-taught Excel certification courses to non-evaluators. Spreadsheet skills can help non-evaluators feel more comfortable with evaluation because it takes some of the mystery out of data analysis.
• I’ve also led brown bags about everything from logic models to research design. As a result, I’ve been more of a data “coach,” guiding staff through evaluation rather than making decisions on their behalf.

Hot Tips: Internal evaluators can use their skills to help their organizations in other ways, including:

• Volunteering at program events. When I served food to child and teen participants at Thanksgiving, my time spent chatting with them helped me design more responsive data collection instruments.
• Contributing to organization-wide research projects, such as looking for patterns in data across the participants that programs serve each year.
• Partnering with graduate interns and external evaluators to conduct more in-depth research on key aspects of the organization.

Cool Trick: Eun Kyeng Baek and SeriaShia Chatters wrote about the Risks in Internal Evaluation. When internal evaluators get wrapped inside internal politics, we can partner with external evaluators like consulting firms, independent consultants, and even graduate interns. Outsider perspectives are valuable and keep things transparent.

Rad Resources:

• Bloggers Week: Priya Small on the Nonprofit Capacity Building Blog
• Myia Welsh and Johanna Morariu on Approaches to Supporting Evaluation Capacity Building
• IC TIG Week: Norma Martinez-Rubin on Straddling Multiple Organizational Cultures and Communication Styles

AEA is celebrating Internal Evaluators TIG Week. The contributions all week come from IE members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

dashboards · data-based decisions · excel · interns · logic models · organizational culture · strategic planning · turnover · utilization-focused evaluation · volunteer

We welcome you to join the Internal Evaluation week! My name is Boris Volkov. I’m Chair of IE TIG, working at the Center for Global Health (Centers for Disease Control and Prevention, Atlanta). All next week you’ll hear from our members, internal evaluators from different fields and organizations will share practical examples and lessons from their work on implementing and promoting internal evaluation.  Here’s just a flavor of what’s coming: reasons to (absolutely!) love internal evaluation; cool tricks to develop your internal evaluator’s brand; the importance of outlier stories in our work; and more!

I would like to thank Josh Joseph (Partnership for Public Service, Washington, DC) for his hard work coordinating this aea365 week. It is our hope that the issues to be highlighted in our blogs will be of interest and use to both internal and external evaluators.

My interest for the last few years has been in the roles that internal evaluators play in their organizations. Being involved with health organizations, I would like to use a metaphor of a “family doctor” to describe what internal evaluators do in organizations. I’ve observed that, for many organizations/programs, program evaluation is not so much rocket science as it is brain surgery… without anesthesia. It is a mysterious, frightening, and potentially painful process with a possibility of complications… and even (program) death. Some evaluation pundits might dislike such an “anthropomorphized” view, but I think we evaluate, diagnose, and provide recommendations to programs and program staff in our organizations much like family doctors do with their patients.

Lesson Learned: Be open with staff about their programs! I remember once having a doctor-surgeon who both irritably and condescendingly refused to explain a recommended surgery to me: “Just leave it to me! You don’t need to worry about it,” he said. Your guess is correct: He was not my doctor any longer. Nobody messes with my health without my awareness and participation! By analogy, evaluation “messes” with a program’s health. Quite understandably, organization members should worry about it. As internal evaluators, we should be genuine and help them to understand and participate in the evaluation process.

Rad Resource: The Hippocratic Oath is an oath historically taken by healthcare professionals swearing to practice medicine ethically. Read it, and you’ll see that it is pertinent to your own practice. Of course, The Program Evaluation Standards and American Evaluation Association Guiding Principles for Evaluators are the equivalents of the Oath and are always Radical Resources for us. My hope is that by following the Evaluator’s Oath, I may “enjoy life and art, be respected while I live and remembered with affection thereafter.”  I wish you all the same!

The American Evaluation Association will be celebrating Internal Evaluators TIG Week. The contributions all next week come from IE members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

health · medicine

Hi, I am Robin Kelly; I work as an internal evaluator for the National Minority AIDS Council, which is a federally funded nongovernmental organization that provides capacity building assistance to community based organizations and health departments that have programs or interventions to address the HIV/AIDS epidemic. I mention it because we work in communities of color. In doing so, attention to culture, be it individual, or organizational, must be given paramount attention.

Also, I live and work in Washington, DC. , a  city of hot summers, temperate falls, unpredictable winters, warm springs and varying political winds all year long. It is also an extraordinarily diverse city.  From the people to the types of nongovernmental organizations that exist here, there are a plethora of cultures.

The phrase cultural intelligence (CQ) is used as we systematically address the evaluation of personal interests and interactions. The abbreviated meaning of the term is the capability to function effectively across various cultural contexts (national, ethnic, organizational, generational, etc.) with heightened awareness of the characteristics of those in that setting, be they organizational culture or individual.  Some have referred to cultural intelligence as the sister to emotional intelligence.

Hot Tip: When you are placed in an international or local setting or working with foreign nationals, remember to flex your CQ skills.  When working with individuals or groups who represent diverse cultures, remember that each participant has four distinct capabilities that you will need to be sensitive to:

1. Drive – motivation
2. Cognition- understanding
3. Strategic outlook-awareness
4.  Action-behavior (in new settings)

Rad Resource: Consider using a tool to optimize your CQ. In addition to the tips above a tool will help to reframe or adjust to those with whom you wish to consult or interact or provide a service.  To see a tool that I recommend, see SAMPLE CQ SELF-TEST.  The complete assessment can be found in Building Cultural Intelligence by Richard D. Bucher.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Capacity Building · cultural competency · cultural intelligence · culture · diversity · minority