Bethany Laursen on How to Evaluate the Situation, Not Just the Program: It’s Complex!

I’m Bethany Laursen, Evaluation Outreach Specialist with the Solid & Hazardous Waste Education Center (SHWEC) at the University of Wisconsin. I’m also principal consultant at Laursen Evaluation and Design, LLC. At SHWEC, I help staff design programs that engage opportunities to achieve our mission. Opportunity hunting requires a form of situation assessment, which has not been widely or deeply discussed in evaluation—especially when it comes to evaluating opportunities in complex, dynamical situations.

Rad Resource: AEA’s EvalTalk and TIG group listservs as peer learning communities.

Through EvalTalk, several colleagues helped me distinguish among three approaches/tools that all claim to be useful in developing programs in complex situations: needs assessment (NA), developmental evaluation (DE), and strengths, weaknesses, opportunities and threats (SWOT) analysis.

Lesson Learned: NA, DE and SWOT are all necessary parts of evaluating complex situations and program responses.

To summarize this discussion so far, we have the following options, where () means “as a part of” e.g. NA is a part of SWOT:

1. NA–>SWOT–>DE
2. SWOT(NA)–>DE
3. NA–>DE(SWOT)
4. DE(NA, SWOT)

Any of these combinations is logical, although #4 might be difficult without one of the others occurring first. What is not logical is leaving one of the triumvirate out (NA, DE, and SWOT). Here’s why:

SWOT is inherently evaluative: it assigns data a certain value label (S, W, O, or T), based on the criteria “effect on our organization’s goals.” Clearly, we need data to do a reality-based SWOT, and this is why we must include a needs assessment. But a NA per se is not going to be enough data, because many clients think a NA is just about external stakeholders’ needs (Os), not internal capacity (Ss and Ws) or larger system realities (often Ts). (If preferred, one could also frame a NA as an ‘asset assessment.’) These external and internal ‘lessons learned’ from our situation should inform developmental program evaluation.

In complex situations, needs assessment is more usefully framed as ongoing situation assessment. This is what I see as the main evaluation task in the Creative Destruction phase of the adaptive cycle. Once we have a lay of the land (situation assessment) and we’ve evaluated the best path to start down (SWOT analysis), then we can jump into developmental evaluation of that path. Of course, what we find along the way might cause us to re-assess our situation and strategy, which is why #4 above is a logical choice.

Lesson Learned: Listen to the language your clients are using to identify relevant evaluation approaches and tools. In SHWEC’s case, our connection to the business sector led me to SWOT analysis, strategic planning, and Lean Six Sigma, all of which are evaluative without necessarily marketing themselves as evaluation approaches.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.