Greetings fellow AEA365 blog readers. My name is Duncan Meyers and I am a graduate student at the University of South Carolina in the Clinical/Community Psychology program. I have a strong interest in evaluation and have been an evaluator for community-based mental health services, an after-school obesity prevention program, and a project aimed at increasing walking through environmental strategies in underserved communities. Within the last three years the projects I have been involved with have utilized geographic information systems (GIS) to enhance our evaluations. A GIS is a system which consists of computer software and data; specifically, geospatial data (i.e., entities or events that can be described in a geographic fashion). These systems can be used to view and manage information about geographic places which are important to your evaluation (e.g., states, counties, communities, schools, etc.) and analyze spatial relationships.

Given the ways in which GIS has benefitted the projects that I have been involved with, I would like to share a helpful tip and point you toward a free resource that I use very frequently.

Hot Tip: First and foremost – like any software assisted analytic tool – GIS involves a learning curve and it is unlikely that evaluators will be able to sit down and use it right away without any training. However, utilizing professional contacts you may have is a great way to work around the learning curve. As long as a clear research question is identified, these contacts may be able to help you visually display your data and analyze relationships among your data and test hypotheses. If you don’t already have such contacts, many county, state, and national agencies/organizations have GIS offices that may be willing to collaborate with you. Also, if there is a University near you the Geography department can be a great help.

Rad Source: One of the most helpful – and basic – functions I have used with GIS is geocoding. Geocoding finds the latitude and longitude coordinates of an address. These coordinates are essentially x and y coordinates that you can put on a map. Here’s a free geocoder that I use a lot: http://www.batchgeo.com/

Batchgeo.com allows you to use an Excel spreadsheet to convert addresses into map coordinates, and then create maps that you can view. Download the template, add some addresses, and hit the “map now” button. Here’s a YouTube tutorial to help get you started: http://www.youtube.com/watch?v=dQAUZqKR2cw

Geocoding is a basic yet integral step in setting up a GIS analysis. Linking the maps you create to your evaluation will take additional steps, and if you are interested in getting some ideas be sure to tune into the AEA Coffee Break Webinar on June 3^rd! http://comm.eval.org/EVAL/coffee_break_webinars/Home/Default.aspx

excel, Geocoding, Geographic information systems, GIS

My name is Matt Keene and I work with the Environmental Protection Agency’s Evaluation Support Division, where I help to coordinate the Environmental Evaluators Network (EEN). The purpose of the EEN is to advance the field of environmental evaluation through more systematic and collective learning. The EEN’s 5^th annual Forum will be held in Washington D.C. June 7-8, 2010.

A hurricane is a powerful and complex system. As it spins and gathers strength out over the ocean, some of its energy is transferred to the water’s surface… stirring it up, causing chaos. But nature seeks stability. The new energy in the ocean organizes into waves that move away from the hurricane’s eye and toward a destination where it will get to work on a rocky point or a sandy beach.

Hot Tip: Join the EEN LinkedIn Group to connect and collaborate. I like to think about the Environmental Evaluators Network as part of the organizing process existing mainly in the space between the eye of the storm and the coast. Demands for evidence of ‘what works’ have stirred things up and the resulting groundswell causes some to float and some to sink though they may be distant from decisions to measure and evaluate. There are many and diverse interests in the space between storm and shore. The EEN facilitates more efficient self-ordering in that space so that we are more systematic and purposeful in channeling information flows and surfacing goals and incentives that sustain a more effective environmental community. Join the EEN LinkedIn Group at http://www.linkedin.com/groups?gid=1773788&trk=myg_ugrp_ovr.

Rad Resource: Have a gander at the website www.nfwf.org/een and agendas, presentations and participants of past and future EEN events. The emphasis on evidence is not exclusive to one country or one organization but is globally disperse, so – naturally – EEN nodes are forming. The U.S. National Oceanic and Atmospheric Administration has joined the U.S. Environmental Protection Agency, National Fish and Wildlife Foundation and The George Washington University Trachtenberg School as co-sponsors. And in addition to the annual Forum in Washington D.C., Environment Canada will host its 3^rd EEN event in September 2010, while planning is underway for 2011 events in Europe, Mexico and the Hawaiian Islands.

Rad Resource: Check out journal editions written and edited by EEN participants.

• New Directions in Evaluations – Environmental Program and Policy Evaluation: New Directions for Evaluation, Summer 2009*
• Evaluation and Program Planning – Challenges in Evaluation of Environmental Education Programs and Policies, May 2010

Cool Tricks: Getting to work as performance requirements pound the coast…

*If you are an American Evaluation Association member, you have free electronic access to all past volumes of New Directions – just sign on to the AEA website using your AEA username and password and navigate over to the journals under the Members Only menu.

conference, Environment, Environmental Program Evaluation, EPE, Linkedin

My name is Karlyn Eckman. I’m a researcher at the University of Minnesota Water Resources Center, and also a long-time United Nations consultant. I’d like to share an evaluation tool that we’ve borrowed from our colleagues in public health and psychology. The knowledge, attitudes and practices (KAP) study method has been used for decades in international health circles. KAP is a highly focused study of people’s knowledge, attitudes and practices surrounding a specific issue or problem. KAPs are done twice, before and after a project activity, and are useful for planning as well as evaluating outcomes. We thought that the KAP method might have some utility for evaluating environmental projects and have successfully tested it in a variety of projects in Minnesota.

We found that the social outcomes of natural resources projects programs are generally overlooked. Do audiences actually adopt and maintain a recommended “best management practice” for water quality? Does knowledge increase as a result of an environmental education program? How do opinions and attitudes change over time about stormwater runoff or invasive species? A major challenge is that the staff of public agencies charged with managing natural resources in Minnesota lack training in social science research or social evaluation methods. Since project staff have limited time and resources to devote to evaluation, we wanted a cost-effective, practical “do-it-yourself” evaluation method that could be learned with training and mentoring.

Rad Resource: The KAP approach seems to fit the bill for documenting specific outcomes and results. To date, we have conducted seven KAP studies on various programs (invasive species; nonpoint source pollution; urban stormwater; septic management; road salt application; shoreland buffers; etc.). KAP has proven to be an adaptive, flexible tool that can be used in multi-disciplinary programs. It can be scaled up or down to accommodate small or large audiences (purposive and random probability sampling). KAP can be used in combination with other research or triangulation methods, such as participant observation. Agency partners indicate that they like the method’s low cost, relative ease of use, and ability to demonstrate social results.

Hot Tips and Cool Tricks: Start the KAP study process by thinking about what you don’t know about your audience, but should. Brainstorm about gaps in your knowledge about your audience that will be needed to evaluate outcomes, and make a list of those gaps. This gap analysis list becomes the basis for building a KAP questionnaire.

Form a small team of three to five people to do the gap analysis, and to develop the KAP questionnaire. This helps to ensure that key pieces of information are not overlooked. Keep the KAP questionnaire short and very focused, ideally under twenty questions.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Attitudes, Environment, Environmental Program Evaluation, KAP

This is Andy Rowe. I am an independent consultant operating from the US in South Carolina and Canada from Hilton Beach Ontario. My first evaluation of a resource related program was an evaluation of the Newfoundland Bait Service in 1985. Since then I have undertaken evaluations in many social and natural settings and on all continents. For the past decade most of my work has been on environmental and conservation efforts in the US and Western Pacific.

The hallmark of evaluation in resource, conservation and environmental settings is that it occurs at the intersection of complex and linked natural and human systems. Broadly speaking there are three programmatic classes of interventions (hence resource, environmental and conservation): resource use is about human use of the natural environment for commercial, recreational, subsistence and ceremonial purposes; conservation is about protecting the natural system from harmful resource use; and environment is generally about improving the state of both natural (e.g. improving water quality) and human (e.g. public health) systems. Evaluation thinking about these settings is still nascent and most evaluations are undertaken by domain specialists with little or no evaluation training or experience.

The mechanisms of change are always found in the human system, usually transmitted through both systems and resulting in changes to both systems. Evaluating two complex intersecting systems is hard. For example, it is hard enough to feasibly and ethically apply experimental and quasi experimental designs in human systems, much harder when one has to control for two complex systems. Likewise, getting salmon to tell the story of their experiences while in the open sea for two to five years is often a challenge.

Hot Tip:

• Engage clients/program officers in identifying the mechanisms of change and discussing sustainability.
• Avoid over simplifying; logic models and related approaches do not easily capture complexity.

Those commissioning evaluations usually acknowledge that both systems have a role, but they are most interested in results in the natural system. They usually have what could be termed a faith based vision of change: for example peer reviewed publications will lead resource managers and governments to change their policies; or that in a world of rapidly declining resources and growing inequality enforcement is a sustainable approach against poaching.

Rad Resource: The work of Westray et al adapting Stacey’s complexity model is a useful framing tool when dealing with two complex systems. Their approach is invaluable as a descriptive frame and for discourse about the location of mechanisms of change. Click here for an illustration from our recent formative evaluation of the David and Lucile Packard Foundation Ecosystem Based Management Initiative.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

complexity, Conservation, Environment, Environmental Program Evaluation, human systems

My name is Jane Peters, and I am the owner of a 19 person firm in Portland Oregon that conducts process and market evaluations of energy efficiency, renewable energy program and other resource conserving programs. I am a 20-year member of AEA and a board member of the International Energy Program Evaluation Conference (IEPEC), which holds a biannual conference on energy program evaluation. I have been doing energy program evaluation since the early 1980s and have seen interest in energy efficiency wax and wane several times. Today, energy efficiency is a top-of-mind issue and evaluation for energy programs is a growth field.

Energy program evaluation functions in a regulatory environment. Regulators set the policies that provide the funds for energy programs and typically require evaluations to assess program saving impacts and make recommendations to improve the program cost effectiveness. Regulators are very busy and most do not have a deep understanding of evaluation, as a consequence many tend to have high expectations for evaluation.

Hot Tip: Outreach and inclusion of regulators increases their understanding of evaluation. The IEPEC does outreach to regulators about IEPEC conferences, provides funding for regulators to attend the conference, sponsors panels at each conference including regulators, and including at least one regulator on the planning committee. This approach brings those who really need evaluations to see how as one of my colleagues says “the sausage is made.” They hear of various methodological issues that contribute to confusing results and learn of programs that are being effective.

This intentional strategy of engagement has been beneficial to evaluators who learn more of the regulatory point of view and to regulators who become more cognizant of the challenges and opportunities of program evaluation.

Hot Tip and Invitation: To learn more about energy program evaluation visit www.iepec.org. For over 25 years energy program evaluators have wrestled with issues of estimation of gross and net savings, how to improve energy programs, how to reach different audiences as well as barriers and opportunities to adoption of energy efficient equipment and behaviors. To help new energy program evaluators be effective, the IEPEC has posted electronic copies of all papers that have been published in conference proceedings since 1999; earlier conference proceedings may be posted in the near future.

Funding for energy efficiency and renewable energy is increasing and is likely to continue to grow in the future. Because this is policy based, opportunities for evaluators to engage with energy programs will expand along with the funding.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

conference, energy, Environment, Environmental Program Evaluation, papers, regulators

My name is Juan Paulo Ramírez and I am a research specialist with the University of Nebraska Public Policy Center. I use Geographic Information Systems (GIS) for project evaluations which have included a broad variety of applications both in the social and physical sciences. Recently a lot of interest has been concentrated on geographic visualization, in particular the integration of geobrowsers like Google Earth with commercial GIS software such as ArcGIS and others. This allows distributing GIS data to many who would not necessarily use GIS software but are consumers of geospatial data. The good news is that the use of GIS software and the data associated to it has become easily accessible to the evaluation community at a very reduced cost and sometimes with no costs at all!

Hot Tip: Check out the YouTube videos posted by ESRI, the California based enterprise that created ArcGIS. Search for “ESRI TV.” If you are a neophyte to GIS and in particular to the ArcGIS family, these tutorial videos will save you thousands of dollars in training. Even if you have experience in using ArcGIS, these tutorials will demonstrate new tricks that will enhance your analytical capabilities. http://www.youtube.com/results?search_query=esri+tv&aq=f

Cool Trick: The newest versions of GIS software, including ArcGIS, have incorporated exporting features for Google Earth format files, known as “KML” format. This will allow you to distribute your GIS information (points, lines or polygons) along with databases to your stakeholders, colleagues, and community in general who do not have GIS software. The only requirement is that your recipients must install the free version of Google Earth in order to read KML format files, and the geobrowser will automatically display all the geospatial information and associated databases that you sent to them (i.e., attached in a e-mail). To see the new features of Google Earth, check: http://www.youtube.com/watch?v=GSuJq4UzkIA

Rad Resource: Want to learn more about GIS and spatial visualization? The following book explores the theory behind geographical visualizations, including examples of map animation, and geovisualization tools, and provides insights to the future development of geographic visualization: Dodge, M., McDerby, M., & Turner, M. (Ed.). (2008). Geographic visualization: Concepts, tools and applications. West Sussex, England: John Wiley & Sons, Ltd.

Want to learn more from Juan Paulo? He is offering a Coffee Break Webinar on Using Google Earth for Evaluation: Applications in Environmental Evaluation and Beyond this Thursday. This is free for AEA members and a paid pass is available for nonmembers. Learn more at http://comm.eval.org/EVAL/coffee_break_webinars/Home/Default.aspx

data visualization, Environmental Program Evaluation, EPE, Geographic information systems, GIS, YouTube

Hello! My name is Annelise Carleton-Hug and I am a principle partner of Trillium Associates a research, evaluation and consulting company serving clients with programs involving the intersections of environment, education and communities.

Hot Tip: As Chair of the American Evaluation Association’s Environmental Program Evaluation (EPE) topical interest group, I invite you to check out the EPE TIG!  Our TIG members are involved in evaluations of conservation biology; environmental policy & management; environmental & conservation education; energy programs; international & global environmental issues (e.g. climate change & international treaties).  As a result of this diversity, our TIG presentations each year at the annual AEA conference cover a wide variety of topics, and we often host off-site field trips to environmental project sites. EPE TIG members will be sharing tips all this week on the AEA365 blog, as well as hosting the coffee break webinar this Thursday, Earth Day.

Rad Resource: The Atlas of Global Conservation - This new book by The Nature Conservancy will be published on Earth Day, April 22, 2010 and is unlike any other atlas.  It contains more than 100 full-color maps and charts depicting information such as where animal populations are concentrated, which species are in imminent danger of extinction, where forests are disappearing more rapidly, and where nature is thriving. Behind each map lies a database, searchable kilometer by kilometer and assembled on a consistent framework so that maps can be compared against one another.  This will truly be a tool that makes global environmental information accessible to scientists, evaluators and concerned citizens.

Rad Resources: Two of my favorite resource sites for tools relevant to environmental education are:

Informalscience.org –  an online community site that strives to support knowledge-sharing, collaboration and the growth of innovation among diverse professionals in the field of informal science education. In addition to being a rich information resource, the site seeks to connect research, practice and evaluation work to a living collection of informal learning projects.  On the site you’ll find links to helpful resources for evaluation, including a variety of evaluation reports on informal science and related topics.

My Environmental Education Evaluation Assistant, “MEERA” – a site devoted to evaluation resources for environmental education programs.  The site is particularly helpful for practitioners of environmental education, thus I find it useful to share with clients to assist them in improving their evaluation capacity.

Hot Tip: Go outside.  Take a hike, ride a bike, climb a tree. Do something to connect with nature – it might help clear your head and provide new inspiration for your work and life.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Conservation, Environmental Program Evaluation, EPE, Online Community, TIG

My name is Susan Kistler and I am the Executive Director of the American Evaluation Association. I contribute each Saturday’s post to the aea365 blog. Last week, I noted that I was writing from the Nonprofit Technology Conference (NTC). This week, I am going to try to tie together learnings from NTC, upcoming events within AEA, and blog posts on aea365.

Hot Tips: At NTC, I had the pleasure of hearing Beth Kanter and Allison Fine speak and learn a bit more about their upcoming book, The Networked Nonprofit, due out in June. Beth Kanter is the CEO of Zoetica, has been a visiting scholar at the Packard Foundation, and is the author of Beth’s Blog (a must-read for those working in or with nonprofit technology). Beth offered great Social Media measurement tips (see Stewart Lee’s post on the aea365 blog this week for more on using social media for evaluators) as well as ideas on improving webinar-based learning that we can put into place on the AEA webinars series. Beth’s co-conspirator, Allison Fine, is well known for her book Momentum: Igniting Social Change in the Connected Age and for her Social Good podcasts for the Chronicle of Philanthropy. What those who know her currently may not know is that she was also the founder of Innovation Network, a nonprofit dedicated to transforming evaluation for social change. Allison is going to give one of the keynote addresses at the 2010 AEA/CDC Summer Evaluation Institute in June where she’ll be talking about Challenges and Opportunities for Measuring Social Media Efforts in a Networked World. I encourage you to come learn from and with her.

Rad Resources: Starting this Sunday, we’ll be celebrating Earth Week with AEA’s Environmental Program Evaluation TIG.  Each day during the coming week, the contributions on aea365 will come from EPE members; we’ll highlight EPE resources on the AEA headlines feed and members can join Juan Paulo Ramirez on Thursday for an online demonstration on using Google Earth for evaluation. Juan Paulo will be using the free version and it is impressive, but if you want more extensive functionality, Google Earth Pro may be the tool for you (see a comparison here). Pro is normally $400 per user, but I learned from the Google outreach reps at NTC that Google offers grants, in the form of free copies of Google Earth Pro to qualifying nonprofits.

Finally, I want to share a short story and do apologize that this will push the blog post over its word limit. At NTC I sat down to breakfast next to a wonderful woman who read my name badge and asked about AEA. I told her a bit about evaluation, evaluators, and the association. She said “wait, you all do the aea365 blog, don’t you!” and went on to share her kudos. Recently, we completed our 100^th post on aea365. Thank you to all who have contributed time, expertise and commentary. And, special thanks go out to John LaVelle who, as our intern and blog manager, has nurtured the blog through its first months even as he prepared for his own nuptials. John, thank you and congratulations!

allison fine, beth kanter, Environmental Program Evaluation, google, institute, philanthropy, social media, stewart lee

We are the Green Team, aka Kim Walker and Karen Truesdale, members of the organizing committee for the 2010 Conference of the Canadian Evaluation Society. Our role is to encourage practices that optimize the environmental impact of this large conference.

Why worry about being “green”? …. during a typical five day conference, 500 attendees will use 62,500 plates, 87,500 napkins, 75,000 cups or glasses and 90,000 cans or bottles. Plus there are all the greenhouse emissions from people traveling to and from the conference and paper and plastic waste from conference handouts. (www.meetgreen.com)

There are many online resources that can help and specialists who can offer estimates in advance and both quantitative and qualitative assessments after the event. Here is just one company that provided this service for the 2010 Winter Olympics in Vancouver, BC (www.offsetters.ca).

Rad Resource: Optimizing the environmental impact of a project such as this large conference involves assessing the potential for both negative and positive impacts. We developed a green conference checklist to help us identify where we can make environmentally friendly choices with our venue, programming and other conference activities. We also considered offsite activities, or secondary effects related to the conference – and have given delegates information on what they can do personally to reduce their carbon footprint and support environmentally sustainable practices during their visit (http://www.vanaqua.org/oceanwise/sustainable-seafood.html).

Our environmental evaluation will include input from the attendees via a post-conference survey, interviews with other members of the organizing committee, and observations made during the conference. Our checklist provides the framework for survey questions and observations.

Including an environmental evaluation component to this year’s conference is particularly apt, given focus of our conference – Going Green, Gold and Global: New Horizons for Evaluation (http://c2010.evaluationcanada.ca). What could be more appropriate than to plan for, and then evaluate green initiatives for our conference!

Rad Resource: One of our keynote speakers is Simon Jackson, founder of the Spirit Bear Youth Coalition. Simon was noted as one of Time Magazine’s 60 Heroes of the Planet and he will be speaking on “The Power of One”. We also have a great lineup of workshops and presentations on evaluation in an environmental context, including:

• Water management system in Costa Rica and its impact of water
• Trends in environmental evaluation in Canada, United States and internationally
• Evaluation of the federal leadership in environmental and energy performance for sustainability
• Water and development and the World Bank support
• Impact oriented evaluations system and the environment
• Participants’ motives and means in environmental evaluations
• Bio-economic models of evaluation of fisheries related programs

Hot Tip: We are expecting an inspiring conference. Join us in beautiful Victoria, British Columbia, Canada from May 2 to May 5 to Go Green, Gold, and Global!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Canada, checklist, conference, Environment, Environmental Program Evaluation