EPE Week: Andy Rowe on Getting the Evaluand Right

This is Andy Rowe. I am an independent consultant operating from the US in South Carolina and Canada from Hilton Beach Ontario. My first evaluation of a resource related program was an evaluation of the Newfoundland Bait Service in 1985. Since then I have undertaken evaluations in many social and natural settings and on all continents. For the past decade most of my work has been on environmental and conservation efforts in the US and Western Pacific.

The hallmark of evaluation in resource, conservation and environmental settings is that it occurs at the intersection of complex and linked natural and human systems. Broadly speaking there are three programmatic classes of interventions (hence resource, environmental and conservation): resource use is about human use of the natural environment for commercial, recreational, subsistence and ceremonial purposes; conservation is about protecting the natural system from harmful resource use; and environment is generally about improving the state of both natural (e.g. improving water quality) and human (e.g. public health) systems. Evaluation thinking about these settings is still nascent and most evaluations are undertaken by domain specialists with little or no evaluation training or experience.

The mechanisms of change are always found in the human system, usually transmitted through both systems and resulting in changes to both systems. Evaluating two complex intersecting systems is hard. For example, it is hard enough to feasibly and ethically apply experimental and quasi experimental designs in human systems, much harder when one has to control for two complex systems. Likewise, getting salmon to tell the story of their experiences while in the open sea for two to five years is often a challenge.

Hot Tip:

• Engage clients/program officers in identifying the mechanisms of change and discussing sustainability.
• Avoid over simplifying; logic models and related approaches do not easily capture complexity.

Those commissioning evaluations usually acknowledge that both systems have a role, but they are most interested in results in the natural system. They usually have what could be termed a faith based vision of change: for example peer reviewed publications will lead resource managers and governments to change their policies; or that in a world of rapidly declining resources and growing inequality enforcement is a sustainable approach against poaching.

Rad Resource: The work of Westray et al adapting Stacey’s complexity model is a useful framing tool when dealing with two complex systems. Their approach is invaluable as a descriptive frame and for discourse about the location of mechanisms of change. Click here for an illustration from our recent formative evaluation of the David and Lucile Packard Foundation Ecosystem Based Management Initiative.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.