AEA365 | A Tip-a-Day by and for Evaluators

TAG | Needs Assessment

Hello, I am Carolyn Cohen, owner of Cohen Research & Evaluation, LLC, based in Seattle Washington. I specialize in program evaluation and strategic learning related to innovations in the social change and education arenas.  I have been infusing elements of Appreciative Inquiry into my work for many years.  Appreciative Inquiry is an asset-based approach, developed by David Cooperrider in the 1980s for use in organizational development. It is more recently applied in evaluation, following the release of Reframing Evaluation through Appreciative Inquiry by Hallie Preskill and Tessie Catsambas in 2006.

 Lessons Learned:

Appreciative Inquiry was originally conceived as a multi-stage process, often requiring a long-term time commitment. This comprehensive approach is called for in certain circumstances. However, in my practice I usually infuse discrete elements of Appreciative Inquiry on a smaller scale.  Following are two examples.

  • Launching a Theory of Change discussion. I preface Theory of Change conversations by leading clients through an abbreviated Appreciative Inquiry process.  This entails a combination of paired interviews and team meetings to:
    • identify peak work-related experiences
    • examine what contributed to those successes
    • categorize the resulting themes.

The experience primes participants to work as a team to study past experiences in  a safe and positive environment. They are then  able to craft  strategies, outcomes and goals. These elements become the cornerstone of developing a Theory of Change or a strategic plan, as well as an evaluation plan.

  • Conducting a needs assessment. Appreciative interviews followed by group discussions are a perfect approach for facilitating organization-wide or community meetings as part of a needs assessment process.   AI methods are  based on respectful  listening to each others stories, and are well-suited for situations where participants don’t know each other, or have little in common.

Using the resources listed below, you will find many more applications for Appreciative Inquiry in your work.

Rad Resources:

The American Evaluation Association is celebrating Best of aea365, an occasional series. The contributions for Best of aea365 are reposts of great blog articles from our earlier years. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hi! I’m Madhawa “Mads” Palihapitiya, Associate Director at the Massachusetts Office of Public Collaboration at UMass Boston. We recently concluded the first phase of a statewide municipal conflict resolution needs assessment study commissioned by the state Legislature.

Hot Tip: The term “need” can mean many things to many people. For needs assessment purposes, needs are defined as Gaps in Results.

Organizations don’t often think of aligning their institutional needs with the societal bottom-line, but working towards this alignment is crucial.  We needed to investigate if our institutional mission, which is to help government and other entities address public conflict, was perfectly aligned with the needs of Massachusetts municipalities and their constituents. This alignment was particularly important to us as a statutory state agency and would add measurable societal value. Overtime, this alignment can also increase the institutional bottom-line.

Rad Resource: Roger Kaufman’s Needs Assessment for Organizational Success. See also Bethany Pearsons’ talk at Evaluation 2014 on the Triple Bottom line.

People don’t usually talk about societal results when they talk about organizational needs. How do we define societal results? We first developed an Ideal Vision that contained a series of societal results and indicators to measure them.

Lesson Learned: We had to resist the impulse to focus on immediate institutional needs like organizational inputs and processes. Imagining an ideal future or vision can tell us where the journey should end.

Rad Resource: Kaufman’s Ideal Vision.

Cool Trick: To help the organization and others being engaged understand the difference between different results, consider developing a visualization like the DoView chart below.

Mads 20 April 2015

Assessing societal results while assessing the institutional bottom-line requires access to valuable data both within and outside of your organization. A Needs Assessment Committee (NAC) was established as the ‘public face’ of the process and to provide advice and guidance on assessment design, participant selection etc.

Cool Trick: Set-up a website to communicate the purpose of your needs assessment. Use social media whenever possible.

There are 351 cities and towns in Massachusetts. Multiple organizations were involved. We had limited resources to collect the data we needed. We had to get creative! A series of regional focus groups and telephone interviews were held. To reach the rest, we launched an online survey.

Hot Tip: Online surveys are a great way to involve more people. Keep survey questions close-ended and completion time to 10-15 minutes. Plan ahead so that you keep the survey open for as long as you can.

Cool Trick: Get creative with survey dissemination by using contact databases, newsletters, list servers, Facebook and twitter. Ask people you know to invite others to take the survey.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on theaea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest toaea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hi, we are Monika Mitra and Lauren Smith from the Disability, Health, and Employment Policy unit in the Center for Health Policy and Research at the University of Massachusetts Medical School.  Our research is focused on health disparities between people with and without disabilities.

Evaluating a Population of People with Disabilities

In collaboration with the Health and Disability Program (HDP) at the Massachusetts Department of Public Health (MDPH), we conducted a health needs assessment of people with disabilities in Massachusetts.  The needs assessment helped us better understand the unmet public health needs and priorities of people with disabilities living in MA.  We learned a tremendous amount in doing this assessment and wanted to share our many lessons learned with the AEA365 readership!

Lessons Learned:

  • 3-Pronged approach

Think about your population and how you can reach people who might be missed by more traditional methodologies:  In order to reach people with disabilities who may not be included in existing health surveys, we used two other approaches to complement data from the MA Behavioral Risk Factor Surveillance System (BRFSS).  They included: an anonymous online survey on the health needs of MA residents with disabilities and interviews with selected members of the MA disability community.

  • Leveraging Partnerships

Think about alternative ways to reach your intended population:  For the online survey, we decided on a snowball sampling method.  This method consists of identifying potential respondents who in turn identify other respondents; it is a particularly useful methodology in populations who are difficult to reach and may generally be excluded from traditional surveys and affect one’s generalizability of findings.  HDP’s Health and Disability Partnership provided a network to spread the survey to people with disabilities, caregivers, advocates, service providers, and friends/family of people with disabilities.

  • Accessibility is Key

Focus on accessibility:  In an effort to increase the accessibility of the survey, Jill Hatcher from DEAF, Inc. developed a captioned vlog (a type of video blog) to inform the Deaf, DeafBlind, Hard of Hearing, and Late-Deafened community about the survey.  In the vlog, she mentioned that anyone could call DEAF, Inc. through videophone if they wanted an English-to-ASL translation of the survey.  Individuals could also respond to the survey via telephone.

Rad Resources:

  • Disability and Health Data System (DHDS)

DHDS is an online tool developed by the CDC providing access to state-level health data about people with disabilities.

  • Health Needs Assessment of People with Disabilities Living in MA, 2013

To access the results of the above-mentioned needs assessment, please contact the Health and Disability Program at MDPH.

  • A Profile of Health Among Massachusetts Residents, 2011

This report published by the MDPH contains information on the health of people with disabilities in Massachusetts.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

· ·

Hello, my name is Hongling Sun. I am a PhD student from University of Illinois at Urbana-Champaign. I feel honored to have this opportunity to share my mixed methods experience with you, with a focus on the lessons and tips I’ve learned through constructing a mixed methods design for a needs assessment study. That design was initially constructed to fulfill the requirement from Dr. Greene’s mixed methods class, and later has been developed into my dissertation study. Below are a few lessons and tips I have learned through this experience.

Hot Tips:

  • Be clear. You cannot claim you will do a mixed methods study before you are clear with what you want to do in your evaluation. A tip here is to remember methods are always subordinate to substantive studies. Therefore, decide your evaluation purposes and questions first, and then decide whether mixed methods is appropriate for your study, and if yes, decide what specific mixed methods purposes and design fit into your study.
  • Use high quality rationale. You cannot justify your use of mixed methods with ‘general’ words by only quoting mixed methods literature. A high quality rationale is to integrate the substance of your specific evaluation purposes and questions with mixed methods literature.
  • Consider your reasons. It is important to explicitly justify your use of mixed methods in your evaluation. Evaluators who adopt a mixed methods design are encouraged to carefully consider your reasons for using mixing methods (or mixed methods purposes), and your design dimensions (e.g., weight of methods, timing) along with what methods will be used and why, and where the “mixing” will take place in your evaluation.
  • Keep your mixed methods design flexible.Given that mixed methods practice is often more complicated than mixed methods theories, evaluators who design a mixed methods study are suggested to consider what an alternative design can be. For example, a mixed methods study with development purpose could become a complementary purpose, because in practice you may find you have no time to analyze the data from an earlier phase as planned; in that case, you could implement different phases at the same time, and then analyze the data after the completion of the entire data collection.
    • Consider your alternative plan when you construct a mixed methods design. The alternative plan can be a different mixed methods design, or a mono-method design.

These lessons and tips are nothing new. However, I still see them as the most critical lessons I’ve ever truly experienced and understood (not only at the theoretical level), and they remain among the most critical principals in my current practice of mixed methods.

The American Evaluation Association is celebrating Mixed Method Evaluation TIG Week. The contributions all week come from MME members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

·

Hello, I am Carolyn Cohen, owner of Cohen Research & Evaluation, LLC, based in Seattle Washington. I specialize in program evaluation and strategic learning related to innovations in the social change and education arenas.  I have been infusing elements of Appreciative Inquiry into my work for many years.  Appreciative Inquiry is an asset-based approach, developed by David Cooperrider in the 1980s for use in organizational development. It is more recently applied in evaluation, following the release of Reframing Evaluation through Appreciative Inquiry by Hallie Preskill and Tessie Catsambas in 2006.

 Lessons Learned:

Appreciative Inquiry was originally conceived as a multi-stage process, often requiring a long-term time commitment. This comprehensive approach is called for in certain circumstances. However, in my practice I usually infuse discrete elements of Appreciative Inquiry on a smaller scale.  Following are two examples.

  • Launching a Theory of Change discussion. I preface Theory of Change conversations by leading clients through an abbreviated Appreciative Inquiry process.  This entails a combination of paired interviews and team meetings to:
    • identify peak work-related experiences
    • examine what contributed to those successes
    • categorize the resulting themes.

The experience primes participants to work as a team to study past experiences in  a safe and positive environment. They are then  able to craft  strategies, outcomes and goals. These elements become the cornerstone of developing a Theory of Change or a strategic plan, as well as an evaluation plan.

  • Conducting a needs assessment. Appreciative interviews followed by group discussions are a perfect approach for facilitating organization-wide or community meetings as part of a needs assessment process.   AI methods are  based on respectful  listening to each other’s stories, and are well-suited for situations where participants don’t know each other, or have little in common.

Using the resources listed below, you will find many more applications for Appreciative Inquiry in your work.

Rad Resources:

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

I’m James Altschuld, Professor Emeritus of Ohio State University. So you have to conduct a needs assessment! You are an evaluator in an organization or expert consultant, you know what to do. You have a sense of what needs assessment is but you really haven’t been involved in one before and now you are concerned.

Lessons Learned: Use the 3 Phase Modelneeds assessment books

One framework that could guide you (from a biased perspective) is the 3 Phase Model in books that I coauthored,  

Planning and Conducting Needs Assessments (1995) and in 2010’s Needs Assessment Toolkit: An Overview (Book 1), described below.

Phase 1 – Pre-Assessment. Form a guiding committee to work on the assessment. It defines concerns, finds existing information on the ‘what should be’ and ‘what is’ conditions of needs, and specifies discrepancies between them.

  • Keep the committee small for efficiency.
  • Select knowledgeable members with the time to locate relevant reports, articles, past studies/needs assessments, etc.
  • Look for available sources rather than creating data.
  • Examine what was found and decide that the needs are not important enough to do anything more, we should explore them in greater depth (Phase 2), or there is enough understanding to move to solutions (Phase 3).

Hot Tip – This is the least costly part of needs assessment because new data is not sought.

Phase 2 – Assessment. The committee requires more information via surveys, interviews, observations, epidemiology, data base exploration; once collected and analyzed there is a better idea of needs – size, causes, which might be of highest priority, and even potential solution strategies for the most important ones.

  • Employ qualitative and quantitative methods for greatest understanding.
  • Scan the literature for methods, instruments, and possible questions.
  • Obtain data from service recipients, service providers, and administrators.

Hot Tips – Collecting, analyzing, and reporting new information are costly. Consider smaller samples and cheaper ways to do it.

Phase 3 – Post Assessment. From the findings in Phases 1 and 2, the committee selects strategies for resolving needs and helps the organization plan and evaluate them.

  • Find out what other organizations have done with similar problems.
  • Break solutions into smaller parts for implementation, if possible.
  • Needs assessors should stay involved with organization for a good portion of Phase 3
  • Evaluating the needs assessment process and solution strategies is in all 3 Phases.

Hot Tip – Solutions flow from the causes of needs.

Hot Tip – Join the Needs Assessment Topical Interest Group leadership at our Business Meeting during the AEA Annual Conference in October to get more involved in the TIG!

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the NA AEA Topical Interest Group. The contributions all this week to aea365 come from our NA  TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hi, my name is Jim Altschuld.  I am a faculty emeritus at the Ohio State University in the school of educational policy and leadership.  Today I will be sharing some tips on needs assessment.

You’re called in and they want to assess their needs.    That’s great because you’ve been steeped in Witkin, Altschuld, Kaufman, Gupta, McKillip, Leigh, Watkins, Wedman, and others – no problem, no sweat let’s get started.   Not so fast!   Hold on a second!

Hot Tip: Most of the time sponsors have vague ideas of what a need is, what it means to conduct an assessment, and what they would do with the results.  So one of your first tasks is to gently question what’s prompting the request and what they understand.  Ask about why your help was sought, what is the nature of problems, how they know about them – where is the information coming from, and lots of probes like this.  Do it softly but get people to open up in this way, for better or worse this is lightly camouflaged reconnaissance.   You’ll probably have to guide and teach them in a subtle not pedantic manner.

Hot Tip: It may be that an assessment is not necessarily the best thing to do!   All that may be required is a quick analysis of already available data and reports or a brief scanning of a few articles and sources.   Indeed, this has been the experience of several of the above authors as noted in the case descriptions below.

  • Case 1 The consultant was brought in to assess needs via a survey that the sponsoring agency felt was best.   After initial interactions it was clear that they would be wasting their money and her time, as the agency did not know what was involved in a survey, wasn’t sure of the direction of the assessment and what questions to ask, and hadn’t in any way explored other methodological options.
  • Case 2 An agency requested a needs assessment but after initial discussions the needs were but not what the agency really needed to focus upon. The external consultant told the agency this, and lost compensation as a result, but gained in professional satisfaction.
  • Case 3 A curriculum planning group sought assistance with needs assessment subsequent to having done review of pertinent literature and analysis of their current offerings.   As it turns out, the group’s only need was guidance in organizing what they had already explored.

In every case the role for the needs assessor, the costs to the sponsoring group, and the formal collection of new data were greatly reduced or totally negated.  The effect of the reconnaissance effort is underscored and is highly recommended before beginning any needs assessment project.   It might lead to less compensated work for the evaluator but it is in keeping with the professionalism of the field and making sure that stakeholders receive maximum benefit from what a skilled and trained evaluator can provide.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

·

Archives

To top