AEA365 | A Tip-a-Day by and for Evaluators

CAT | Needs Assessment

I’m Maurya West Meiers, Senior Evaluation Officer at the World Bank and coauthor of A Guide to Assessing Needs: Essential Tools for Collecting Information, Making Decisions, and Achieving Development Results.

I often work with groups in carrying out needs assessments, collecting data, training, facilitating retreats, etc.  So I’m always looking for facilitation tips and resources.  Today I’m sharing some favorites.

Lessons learned: 

If your end-goal in your meeting with a group is to gather data or make decisions (through focus groups, multi-criteria analysis, etc.), you’ll want to do some early rapport building to get people comfortable with one another and talking.

  • Make sure the right people are in the room. It seems obvious, but take the time to define your targets in advance and make sure that those participating are those targeted.  Be prepared to gently remove people who don’t fit your pre-defined needs.  Have another coordinator with you to help in this process.  And have your room comfortably furnished and arranged.
  • Learn the names of participants in advance and give a warm greeting when they enter.  These are common networking techniques because they work and put people at ease.
  • Use name badges and table tents. Have these items ready.  You may wish to let participants write their own names instead of pre-printing them.  Perhaps Jennifer prefers to have everyone call her Jen – so give her the chance to write her name as she wishes.
  • Get people talking early.  As people enter the room, introduce them to others – and have ideas listed on a flip chart or card that they can discuss with one another.  Keep people moving and mixing.  Use a chime or bell to signal a move.

Use icebreakers.  An easy icebreaker involves giving participants name badges and asking them to write two or three things they feel comfortable discussing with others.  Example:

Meiers

  • Energizers and games. If your group work – such as in a retreat – covers a lengthy period of time, use energizers (usually involving some movement) or games to keep people alert and engaged.  If you search for energizers on YouTube, you’ll find many ideas you can adopt and adapt for your purposes and you’ll see how they work ‘in action’ and not just on paper.  This quick and easy energizer is one such example.

Rad Resources. Here are some of my “go to” books and websites on facilitation techniques and tools.

 

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

No tags

Hello, my name is Sue Hamann, and I work at the National Institutes of Health. One part of my job is to solicit and review proposals from evaluation contractors for various types of evaluation projects, including needs assessment and program planning. Today I will share some tips about improving your proposed needs assessment.

Hot Tips:

Show that you are knowledgeable about needs assessment. Don’t even think about submitting a proposal that does not define needs assessment or mention its rich history and development. This sounds obvious, but you would be surprised that I often review proposals that are lacking evidence that the proposers know the field. Be sure to cite the Altschuld and Ryan edition (New Directions for Evaluation, #144, winter 2014). Justify your proposed methodology based on existing literature.

Assemble and describe a team with all the required skills. Michael Scriven in his 1991 Evaluation Thesaurus listed the following as content areas in which evaluators should be skilled: statistics, cost analysis, ethical analysis, management theory and practice, pedagogy, social psychology, contract law, interviewing skills, professional politics, presentation graphics, dissemination, synthesis. Altschuld and Watkins (2014) stated that needs assessment involves the following methods, in addition to the qualitative and quantitative methods employed in other evaluation activities: gap analysis, causal analysis, prioritization strategies, comparison of solutions. Needs assessment is usually a team effort. Make sure that you document and budget for the skills that you already have available and the skills that you will add to your team.

Be alert to the culture of the organization to which you are applying. I read a lot of proposals that are obviously boilerplate. It is generally not worth your time to submit a vague needs assessment proposal, that is, one that is not tailored explicitly to the organization or solicitation. Sometimes you have to do some online searching about the history and status of an organization to determine the kind of needs assessment that will be consistent with the culture of an organization and useful to it. Be sure to read the article from Maurya West Meiers and colleagues (New Directions for Evaluation, #144, winter 2014); it has great tips about planning international needs assessments, but the tips are applicable to any new environment.

Document your membership in AEA. If you are reading this blog, you are probably a member of the largest group of evaluators in the world. Be sure to mention this when you state your qualifications. If you belong to the Needs Assessment TIG, say so. If you do not belong and you are interested being paid to do needs assessments, you should join and become active in the TIG.

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Ever thought about doing research on needs assessment (or evaluation for that matter)? My name is Ryan Watkins and what follows is a short description of a basic framework that can help you consider the types of research that could be useful. Based on Briggs (1982) and Driscoll (1995), the following research paradigms can offer choices for your research design:

Experimental and Quasi-Experimental: Experimental research designs can provide educational researchers with the “most effective means of establishing causal influences on a phenomenon of interest” (Driscoll, 1995, p. 323). While quasi-experimental research designs may be less effective in providing evidence of casual relationships, they offer pragmatic alternatives when meeting the requirements of experimental designs are not practical.

Meta -Analysis: The meta-analysis research design is “a widely used method for combining results from different quantitative research studies” (p. 71) on the same phenomenon (Gall, Gall, and Borg, 1999).

Case Study/Ethnography: According to Trochim (2001), “A case study is an intensive study of a specific individual or specific context… There is no single way to conduct a case study, and a combination of methods (such as unstructured interviewing and direct observation) can be uses.” (p. 161)

Technology Development and Evaluation using a Novel Technique: As a field of research focused on useful application, needs assessment research should be active in the development, application, evaluation, and continuous improvement of assessments using technologies (e.g., online surveys).

Cost Analysis: The requirements of decision-makers for data related to the cost-effectiveness of decisions presents researchers with a pragmatic research paradigm with obvious application benefits.

Model Development and Evaluation: Models offer useful tools for conceptualizing the relationships and complexities among the components of a system.

Novel Technique Development and Evaluation: Techniques are the processes used to accomplish results (i.e., produce products, obtain outputs, and/or achieve consequences). The development and validation of a novel technique for accomplishing results within needs assessment (or evaluation) is an essential role of the researcher.

Theory Development: Theories help us explain or predict a phenomenon. In others words, a theory is the answer to why something (such as a behavior) occurs or does not occur within a context. Making and testing theories helps advance research and practice.

Rad Resources:

Briggs, L. (1982). A Comment on the Training of Students in Instructional Design. Educational Technology. 22(8), 25-27.

Driscoll, M. (1995). Paradigms For Research in Instructional Systems. In Angling, G. (1995). Instructional Technology: Past, present, and future. Englewood, Co: Libraries Unlimited.

Gall, J., Gall, M. and Borg, W. (1999). Applying Educational Research: A practical guide (4th ed.). New York: Addison Wesley Longman.

Trochim, W. (2001). The Research Methods Knowledge Base (2nd ed.). Cincinnati, OH: Atomicdog.com Publishing.

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I’m James W. Altschuld. When Ryan Watkins asked me to write a NA entry for this year’s aea365 sponsored week, he playfully dubbed me the ‘Patron Saint’ of Needs Assessment. While his perception is not accurate, allow me in that spirit to offer some commandments for Needs Assessors.

Thou:

  1. art a facilitator of a process and never a savior which thou wilt never be
  1. shalt always explore the organization or group thou art working with for existing information before collecting any new data
  1. shalt have an advisory or guiding group built into the needs assessment process for eventually its members will probably be part of the needs resolution strategy
  1. must assume that many voices will enter the fray increasing its difficulty
  1. must expect frustration when doing needs assessments (art thou surprised for it comes with the territory, no elaboration is required, and it fits with commandments 3 and 4)
  1. wilt consult the literature for ideas and input as painful as that might be
  1. shalt honor the needs assessment gurus (a nutty commandment if there ever was one)
  1. wilt have to allot more time and lucre for the process than thou originally planned (so what’s new)
  1. must recognize that thou art a facilitator or a catalyst but that other not thou have responsibilities for solutions and their implementation
  2. will absolutely adhere to the prior 9 commandments

 

Author’s Note:   After reading the commandments it should be apparent that Dr. Watkins was completely incorrect in applying the Patron Saint designation to this author. Nothing could be further from the truth.

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I’m Bethany Laursen, Evaluation Outreach Specialist with the Solid & Hazardous Waste Education Center (SHWEC) at the University of Wisconsin. I’m also principal consultant at Laursen Evaluation and Design, LLC. At SHWEC, I help staff design programs that engage opportunities to achieve our mission. Opportunity hunting requires a form of situation assessment, which has not been widely or deeply discussed in evaluation—especially when it comes to evaluating opportunities in complex, dynamical situations.

Rad Resource: AEA’s EvalTalk and TIG group listservs as peer learning communities.

Through EvalTalk, several colleagues helped me distinguish among three approaches/tools that all claim to be useful in developing programs in complex situations: needs assessment (NA), developmental evaluation (DE), and strengths, weaknesses, opportunities and threats (SWOT) analysis.

Lesson Learned: NA, DE and SWOT are all necessary parts of evaluating complex situations and program responses.

To summarize this discussion so far, we have the following options, where () means “as a part of” e.g. NA is a part of SWOT:

  1. NA–>SWOT–>DE
  2. SWOT(NA)–>DE
  3. NA–>DE(SWOT)
  4. DE(NA, SWOT)

Any of these combinations is logical, although #4 might be difficult without one of the others occurring first. What is not logical is leaving one of the triumvirate out (NA, DE, and SWOT). Here’s why:

SWOT is inherently evaluative: it assigns data a certain value label (S, W, O, or T), based on the criteria “effect on our organization’s goals.” Clearly, we need data to do a reality-based SWOT, and this is why we must include a needs assessment. But a NA per se is not going to be enough data, because many clients think a NA is just about external stakeholders’ needs (Os), not internal capacity (Ss and Ws) or larger system realities (often Ts). (If preferred, one could also frame a NA as an ‘asset assessment.’) These external and internal ‘lessons learned’ from our situation should inform developmental program evaluation.

In complex situations, needs assessment is more usefully framed as ongoing situation assessment. This is what I see as the main evaluation task in the Creative Destruction phase of the adaptive cycle. Once we have a lay of the land (situation assessment) and we’ve evaluated the best path to start down (SWOT analysis), then we can jump into developmental evaluation of that path. Of course, what we find along the way might cause us to re-assess our situation and strategy, which is why #4 above is a logical choice.

Lesson Learned: Listen to the language your clients are using to identify relevant evaluation approaches and tools. In SHWEC’s case, our connection to the business sector led me to SWOT analysis, strategic planning, and Lean Six Sigma, all of which are evaluative without necessarily marketing themselves as evaluation approaches.

Figure 1: Augmenting a traditional logic model, this is a metaphorical picture of how SHWEC understands our complex, dynamical situation and our potential evaluation questions. (Each sailboat is a staff member.) Next, I had to find evaluation approaches that would fit.

Figure 1: Augmenting a traditional logic model, this is a metaphorical picture of how SHWEC understands our complex, dynamical situation and our potential evaluation questions. (Each sailboat is a staff member.) Next, I had to find evaluation approaches that would fit.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

I’m Lisle Hites, Director of the Evaluation and Assessment Unit (EAU) at the University of Alabama at Birmingham (UAB). I’m writing to share my team’s experiences in conducting needs assessments.

We frequently have opportunities to work with our colleagues on campus to conduct needs assessments for grant-funded projects. One such example was a training grant through the School of Nursing, and we provide it to highlight the value of gathering more than one perspective in assessing needs.

In 2012, CDC data revealed that the South is the epicenter of new infections of HIV; compared to other regions, 46% of all new infections occurred in the region, with a higher percentage of women (24%) and African-Americans (58%) represented in the new infections. Therefore, it is critically important that healthcare providers receive HIV/AIDS training in order to provide HIV/AIDS primary care to meet current and future healthcare demands.

To establish workforce training capacity, we sent surveys to two key healthcare audiences: (1) potential training sites (Ryan White Grantees) and (2) future family nurse practitioners (FNPs). Responses identified both a shortage of trained HIV/AIDS healthcare providers as well as an interest by providers and students to establish clinical training opportunities. Additionally, 78% of current FNP students enrolled at one research institution in the south resided within 60 miles of a Ryan White Grantee site in a tri-state region.

Lessons Learned:

  • The design of this needs assessment allowed us to consider the capacity of Ryan White Grantee sites to provide clinical training opportunities for FNP students.
  • The survey captured the interest and desire of FNP students to seek the skills necessary to provide HIV/AIDS primary care.

Despite the current and future needs for a trained healthcare workforce, healthcare providers in the Deep South still encounter many of the same attitudes toward people living with HIV/AIDS as were found in the early years of the epidemic; therefore, it was necessary to identify a pool of potential candidates for training (i.e., FNP students). At the same time, little was known regarding the capacity and willingness of Ryan White Grantee sites to provide an adequate number of opportunities to meet the training needs of these students. By considering both sides of the equation, we could accurately match the number of students and training sites to ensure a high degree of satisfaction and success for both parties.

Rad Resources: 

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

I’m James Altschuld, Professor Emeritus of Ohio State University. I’ve written a lot over the years about needs assessment.  Today’s posting is about having hybrid vigor in how we approach our work.  It’s not just needs assessment or asset/capacity building, it’s both!

My premise is that there are two contrasting stances.  One is building from strengths, resources, and assets (positives).  The other is from a negative (something is missing) needs perspective.  And yet these stances are eternally interdependent, and share enough common ground that we should commonly assess both.

Lessons Learned From Experience:

Observations

  1. Mind the philosophies, they are different, use unique methods, and the improvement plans could be quite distinct.  One is the glass half full and the other half empty; and the reality is that the glass is both half-full and half-empty.
  1. Hybrid asset/capacity building and needs assessment approaches are now appearing in the literature (health, community development, governmental activities, and related areas).  I’ve found examples of implementation in Scotland, Indonesia, Spain, Minnesota, and elsewhere.
  1. Hybrid assessments always begin from assets to avoid a possible negative taint of needs.
  1. Hybrids require more cost, time, facilitation, coordination, and management than traditional needs assessment.
  1. The voice of the people (the bottom up) is much more prominent in asset/capacity building and hence in hybrid applications.
  1. For intractable problems (health, violence, etc.) hybrids are thought to be better than either needs assessment or asset/capacity building by themselves

Some Implementation Ideas

  1. Use two working groups so that needs and assets can be looked at independently, not contaminated by the other before comparing what is found for each.
  1. Expect hybrids to take longer to complete, just build in more time.

Rad Resources:

Coming in Fall 2014, Ryan Watkins and I have edited an issue of New Directions in Evaluation dedicated to Needs Assessment.

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

My name is Hsin-Ling (Sonya) Hung, an assistant professor in the Department of Educational Foundations and Research at the University of North Dakota and Program Co-chair for the Needs Assessment (NA) TIG. The main concern I want to share in this blog is that we need to pay more attention to the influence of decision-making in NA work.

Lessons Learned Through Experience:

Decision-making is an essential part of the NA endeavor

  • There are many decisions in any NA endeavor. Examples of decisions required relate to the type of needs to be investigated, the selection of methods for data collection, identification of discrepancies, allocation of resources, development of action plans, and others.

Early decision-making has influence on the NA process

  • Decisions made in the very beginning determine how the NA would be conducted. Using the three-phase model (Altschuld & Kumar, 2010) as a starting point, you must decide how to go through all three phrases (pre-assessment, assessment, and post-assessment). Sometimes you might minimize or leave out a phase, affecting such things as methods of data collection, personnel, time, and budget.

Decision-making has impact on NA results or quality of NA

  • Decision-making has an effect on the overall outcome/quality of NA which in turn may change the program/organization funding the endeavor. One of my students worked with a campus organization to identify graduate students’ health needs. A comprehensive approach was discussed initially, but could not be carried out due to funding concerns of the sponsor. Under pressure to get something quickly, the student was only able to collect data via an online survey near the end of the semester. Its timing resulted in small n’s because most students were busy with school work. The quality of the survey was in question because of the press for quick results. Data were not meaningful and participant representation was a concern. Thus decisions in the actual process of NA have an impact on quality and what it produced.

Rad Resources:

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello, my name is Sue Hamann. I work at the National Institutes of Health as a Science Evaluation Officer, and I teach program evaluation to graduate students. Today I’m providing tips to novices in needs assessment (NA).

Hot Tips:

Use the original definition of needs.

  • The original definition of NA is the measurement of the difference between currently observed outcomes and future desired outcomes, that is, the difference between “what is” and “what should be.” Novices often plan to address either status or desired future, but they do not realize how much more valuable it is to collect data about both status and future and analyze the difference between these two conditions. Read anything about NA written by Roger Kaufman, Belle Ruth Witkin, James Altschuld, or Ryan Watkins to get started.

Collect data using multiple methods.

  • A rewarding and challenging aspect of needs assessment is that an evaluator gets to take almost all her tools out of the toolbox. From census data and epidemiologic data to document reviews to group and individual interviews, needs assessment typically requires multiple methods. The best way to start is to review the literature, both in the problem area of interest and in the evaluation journals. You can start with the New Directions for Evaluation issue (#138, summer 2013) on Mixed Methods and Credibility of Evidence in Evaluation, edited by Mertens and Hesse-Biber. Also use listservs such as AEA’s Evaltalk to discover work that has been done but not published.

Keep an open mind about the validity of qualitative data, particularly interviews.

Remember that needs assessment and program planning go hand in hand.

  • Collecting needs assessment data is just the first step in program planning. Use Jim Altschuld’s Needs Assessment Kitor other resources to plan for the work needed to conduct this vital component of program planning and evaluation.

Rad Resources:

Coming in Fall 2014, Jim Altschuld and Ryan Watkins are editing an issue of New Directions in Evaluation dedicated to Needs Assessment.

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I’m Ryan Watkins, Associate Professor at George Washington University and manager of two websites, www.WeShareScience.com and www.NeedsAssessment.org.

Needs assessments require planning. You must plan for the “who, what, why, where, when, and how” of each step – from soliciting participation of stakeholder groups to the iterative development of grounded recommendations.   An overlooked planning task involves the planning for the actual measurement of needs. Below are considerations to guide your planning:

Lessons Learned From Experience:

Consideration 1: What is a need? Needs can be defined in many ways. While discrepancy definitions are common (such as the gap between “what is” and “what should be”), varied definitions are frequently applied (such as needs as desired resources or programs).

Ask:

  • Have you agreed with stakeholders on a definition of need?
  • Are you assessing needs at the state, local, institutional, and/or individual level?
  • Are needs exclusively related to results (most useful), or do they include processes and inputs as well (not useful)?
  • Are needs to be assessed along with assets?

Consideration 2: What data is really required?

When you know how needs are to be defined, next determine what data is required to document the needs.

Ask:

  • What indicators would suggest what needs exist?
  • How could the size and scope of identified needs be measured?
  • Who else is collecting data on similar issues?
  • Which measures are “nice to have” but not absolutely required?
  • When can indirect measures be applied?
  • If applying a discrepancy definition, what data is required for measuring the current state, and what is required for comparably defining the desired state.

Consideration 3: What is feasible?

There are always constraints to a needs assessment and these will guide what is feasible in terms of measuring needs.

Ask:

  • What techniques can be used to collect data?
  • What is appropriate timeframe for measuring (e.g., weekly, monthly, bi-monthly, # of weeks/months after critical interventions)?
  • What is the appropriate sequence for measuring (e.g., indicators #1 and #3, followed by indicator #2)?
  • What resources (people, time, money, technology, access, etc.) are readily available?

Consideration 4: How will it be managed?

Measuring needs is a process that must be managed. Take time to assign responsibilities and hold people accountable for results.

Ask:

  • Who specifically (individuals or organizations) will provide necessary information (e.g., who is the sample, or who has the desired information)?
  • Who is responsible for collecting and analyzing data to measure needs?
  • Who is accountable for the validity of data?
  • Who is going to interpret the data in order to make recommendations?

Rad Resources: Find and share needs assessment resources at www.NeedsAssessment.org, including lesson learned videos, free publications, podcast interviews, and a new document repository.

Clipped from http://needsassessment.org/

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

No tags

<< Latest posts

Older posts >>

Archives

To top