AEA365 | A Tip-a-Day by and for Evaluators

Hi everyone!  I’m Yvonne M. Watson, an Evaluator in U.S. EPA’s Evaluation Support Division and Chair of AEA’s Environmental Program Evaluation Topical Interest Group.  As we celebrate Earth Week in April and prepare for the annual American Evaluation Association  (AEA) conference in October, the theme of sustainability looms large.

As I think about an area where organizations and individuals can make a significant difference to ensure a sustainable future, consumer choice and green purchasing/procurement comes to mind.  The federal government’s role as the leading purchaser of green products is vital to ensuring a sustainable future.  Equally important is the role that households and individuals play in this equation.

Lesson Learned: According to Fischer’s 2010 report, Green Procurement: Overview and Issues for Congress, at the institutional level, federal government procurement accounts for $500 billion annually. Because of its size and purchasing power, the federal government influence on the market is broad—“affecting manufacturing (product planning and development), and purchasing (large institutions and States that mimic federal specifications) both nationally, and internationally.  Established in 1993, the purpose of EPA’s Environmentally Preferable Purchasing (EPP) Program is to: 1) achieve dramatic reductions in the environmental footprint of federal purchasing through creation of guidelines, tools, recognition programs, environmental standards and other incentives and requirements, and (b) make the overall consumer marketplace more sustainable through federal leadership.  In 2011, the EPP program initiated an evaluation to examine the changes in spending on green products across the federal government since 2001. The results indicate greater awareness and positive attitudes towards green procurement among federal purchasers surveyed.

At the individual level, consumers not only vote with their feet – they vote with their purses and wallets too, through the purchase of food, cars, electronics, clothes and a host of other services. In addition, the prominence of green and eco-labels is a prime example of the manufacturing industry’s response to greater demand from consumers who look for green products.  During Earth Week, I encourage organizations, individuals and evaluators alike to take a step back and assess our individual and collective consumer purchasing decisions and the implications for a sustainable future.  After all, the purchasing choices we make today affect the future we have tomorrow.

Rad Resources: EPA’s Greener Products website provides information for consumers, manufacturers and institutional purchasers related to green products.

The EPP Evaluation Report is available here.

The American Evaluation Association is celebrating Environmental Program Evaluation TIG Week with our colleagues in the Environmental Program Evaluation Topical Interest Group. The contributions all this week to aea365 come from our EPE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Andy Rowe here, I evaluate sustainable development and natural resource interventions.  I am convinced evaluation is facing a key adapt or decline juncture.

Connectivity is the mechanism enabling us to understand how interventions reach to the public interest and effects in the natural system. Our siloed governance approaches come from cost and accountability structures in the for-profit sector.  For-profits recognize the importance of connections to the larger mission and judges performance accordingly; now in the mission includes sustainability.  Major corporations such as Mars and WalMart are acting decisively to ensure sustainable supply chains, which they judge essential to survival of their businesses.  We need to begin the process of incorporating sustainability into evaluation.

The story of how domesticated cats contribute to climate change illustrates how obscure but important these causal connections can be.

RoweLesson Learned: Domesticated cats living with humans, and feral cats are a significant predator of songbirds taking an estimated 40% annually.  Birds carry a parasite Toxoplasma gondii. The unharmful parasite departs in stools, often in litter, which ends up in landfills. Landfills are often connected to the sea via groundwater and streams and the parasites enter coastal waters where bivalves ingest them.  Sea otters love bivalves ingesting the parasite, which attacks the otter brains.  Poor otters.

Another system connects with our story. Fertilizer and waste from sewage treatment and other sources deliver nutrients to the sea causing algal growth in the water that weaken sea grasses.  Otters address the effects of excessive nutrient loading on grasses keeping the sea grasses alive.  Sea grasses are amazingly effective at storing carbon – with the help of otters Pacific sea grasses store the equivalent of annual carbon dioxide emissions from 3 to 6 million cars.

So, cats contribute to climate change via mechanisms that are far from transparent.  As evaluators we need to attend to the connections from the intervention to important effects, including effects in the natural system.  By tracing connectivity within and across systems, evaluation can play an important role in ensuring that interventions are designed and undertaken so that the world we leave for our grandchildren is at least as good as the world we inherited.  It is time that sustainability becomes an expected element in evaluation.  Several years ago the National Academy of Science gave sustainability science a room of its own –time now for sustainability to become a required element in our Standards.

Lesson Learned:  Take a look at sustainability in the for-profit sector:  1. Mars Corporation here and here and 2. Walmart here.

Rad Resources:  Otters and weeds:

Also, see Sustainability Science Room of Its Own by William C. Clark (2007).

The American Evaluation Association is celebrating Environmental Program Evaluation TIG Week with our colleagues in the Environmental Program Evaluation Topical Interest Group. The contributions all this week to aea365 come from our EPE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi all! I’m Juha Uitto, Deputy Director of the Independent Evaluation Office of the United Nations Development Programme (UNDP). I’ve spent many years evaluating environment and development in international organizations, like UNDP and the Global Environment Facility (GEF).

As we all know, evaluating sustainability is not easy or simple. Sustainability as a concept and construct is complex. It is by definition multidimensional encompassing environmental, social, cultural, political and economic dimensions. It cannot be evaluated from a single point of view or as just one dimension of a programme. Apart from the above considerations, sustainability refers to whether the programme or intervention that is the evaluand is in itself sustainable. Sustainability evaluation, must take all of the above into account.

At its simplest, sustainability evaluation would look into whether the intervention would ‘do no harm’ when it comes to the various environmental, social, cultural and other dimensions that may or may not be the main target of the programme. At this level, the evaluation does little more than ensuring that safeguards are in place. The evaluation also has to look at whether the intervention itself was sustainable, i.e. whether it has developed exit strategies so benefits will continue beyond the life of the intervention.

But this is not enough. It is essential for evaluations and evaluators to be concerned with whether the evaluand makes a positive difference and whether it has unintended consequences. In environment and development evaluation a micro-macro paradox is recognized: evaluations show that many individual projects are performing well and achieving their stated goals; yet the overall trends are downward. There are lots of projects focused on protected areas and biodiversity conservation; still, we are facing one of the most severe species extinction crises ever. Many projects successfully address climate change mitigation in various sectors ranging from industry to transportation to energy; still, the global greenhouse gas emissions continue their rising trend. It is not enough for evaluators to focus on ascertaining that processes, activities, outputs and immediate outcomes are achieved.

Lessons learned: In evaluating environment and poverty linkages, one should never underestimate the silo effect. Sustainable development requires a holistic perspective but few organizations operate that way. People have their own responsibilities, priority areas, disciplinary perspectives, partners, networks, and accountabilities that often preclude taking a holistic perspective. Evaluators must rise above such divisions. An evaluation – such as the Evaluation of UNDP Contributions to Environmental Management for Poverty Reduction – can make a major contribution to how an organization acknowledges, encourages and rewards intersectoral and transdisciplinary cooperation.

Rad resource: All UNDP evaluation reports and management responses to them are available on a publicly accessible website, the Evaluation Resources Centre, and independent evaluations at Independent Evaluation Office of UNDP.

The American Evaluation Association is celebrating Environmental Program Evaluation TIG Week with our colleagues in the Environmental Program Evaluation Topical Interest Group. The contributions all this week to aea365 come from our EPE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

My name is Sara El Choufi and I wanted to share with you some of thoughts on evaluating the effectiveness aid to the environment.

As evaluators, we tend to focus our work on programs and projects. We thoroughly evaluate a project, or a set of projects and draw out conclusion, best practices, lessons learned, etc. But, I wonder if we ever take a step back and take a look at the bigger picture. I mean really take a step back and try to figure out what the world has achieved in terms of environmental protection in over four decades.

Of course such a study is not an easy task to undertake; for starters, where do we get the data? How reliable is it? Assuming we do have remarkably detailed and reliable accounts, how can we generalize and draw conclusions? To what degree do we rely on quantitative studies, and how much thematic and qualitative work needs to be done?

Thinking about this lead me to Greening Aid? – a book solely focused on the foreign assistance and its impact on the environment. I also discovered what could be considered the most comprehensive database for foreign aid – AidData. Collecting data from the OECD, donors, and recipients, AidData “aimed to create a database of development finance activities with as much descriptive detail as possible at the project level for use in the research community.”

Ok, so we have data, now what? How does one begin to evaluate the impact of aid to the environment on the protection and conservation of the global commons, or forests for example? What about measuring to what degree aid has contributed to the reduction of CO2 emissions? What about marine ecosystems? The list goes on…

Another layer is what indicators do we use? Are the World Development Indicators enough? Do we rely on locally assembled data (be it from government, research institutions, or civil society)? Do we need to have boots on the ground and do our own data collection? So on and so forth…

This seems like an impossible undertaking, or at least an impractical one. Should it be done? How can we as evaluators contribute to such a study?

This is meant as a thought piece, and I hope it compels you to respond and weigh in :)

Rad Resources: AidData – “a research and innovation lab making information on development finance more accessible and actionable.”

Greening aid? : Understanding the environmental impact of development assistance by R.L. Hicks (2010).

The American Evaluation Association is celebrating Environmental Program Evaluation TIG Week with our colleagues in the Environmental Program Evaluation Topical Interest Group. The contributions all this week to aea365 come from our EPE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

 

No tags

Hello, my name is Dan McDonnell and I am a Community Manager for the American Evaluation Association. My last couple of posts have been focused around evaluating Tweets and Twitter data as well as sharing useful Twitter tools for evaluators. This week, I’d like to talk about an upcoming change to Twitter to keep an eye on.

In the coming weeks, Twitter will be giving users’ profile pages a major facelift. A small group of Twitter users, including First Lady Michelle Obama, already have the new page enabled. Besides just looking sharp, the new and improved Twitter profile page adds a number of new features that can come in very handy for users.

Rad Resource: Show Off Your Top Tweets

One of the biggest features that Twitter is advertising is the new ability to display a list of your top tweets prominently. Imagine a fellow evaluation professional who is not currently following you sees you mentioned in a Tweet, and clicks through to your profile to learn more. If they did this today, they’d see whatever your most recent tweet was – whether it was sharing a tip on #dataviz, or checking in at a local eatery. Post update, your tweets that were most engaging (whether retweeted, favorited or responded to) will feature prominently – which helps you put your best foot forward on Twitter.

New Twitter Profile Page

The New Look and Feel

Rad Resource: Pin a Favorite Tweet

Have a tweet that you feel really represents YOU, or the reason that you’re on Twitter? On your new Twitter profile page, you can select your favorite tweet that you’ve sent, and wear it like a badge of honor. Like the above feature, this opportunity allows for you to really customize the first meeting that many Twitter users will have with you, helping to make an introduction on your terms.  See how the band Weezer has taken advantage of this on their new page.

There are a couple more features that the new pages offer, as well as some changing guidelines for the design of your Twitter page – but we’ll save those for a future post. For now, start sifting through your Tweets to find your favorite, and keep your eyes peeled for the update!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Dan McDonnell is a regular Saturday contributor to AEA365, where he blogs on social media-related topics for evaluators. You can reach Dan on Twitter at @Dan_McD.

·

I’m David Fetterman, evaluator, author, entrepreneur, and Google Glass user. Yesterday, we talked about what Google Glass is and how it can revolutionize communications. Today, let’s turn to thinking about how Glass could be used as an evaluation tool.

David Fetterman's Son

Hot Tips – Glass for Empowerment Evaluation: Youth (with parental permission) can wear the Glass to produce photovoice productions, sharing their pictures of their neighborhoods and videos of the activities. It’s easy (and fun) – that’s my son over on the right trying out Glass. Their stories can be used as part of their self-assessment, gaining insight into their lives and potentially transforming their worlds.

Community and staff members can post their digital photographs (and videos) on a common server or blog while conducting their self-assessment with the blink of an eye. This ensures community access, a sense of immediacy, and transparency.

Community and staff members can use Google Hangout on Glass to communicate with each other about their ratings, preliminary findings, and plans for the future.

Hot Tips – Glass for Traditional Evaluation: Evaluators can use it to communicate with colleagues on the fly, share data (including pictures and video) with team members, and conduct spontaneous videoconference team meetings. Note that everyone doesn’t need to have Glass, as Glass users can leverage its capabilities while connecting with others who are using Smartphones or computers.

Glass stamp dates photos, videos, and correspondence, ensuring historical accuracy.

Glass can be used as an effective “ice breaker” to gain access to a new group.

Evaluators can also solicit feedback from colleagues about their performance, with brief videos of their data collection and reporting behavior. There is a precedent for this type of critique – assessments of student teaching videos.

Glass can be used to provide “on the fly” professional development with streaming video of onsite demonstrations for colleagues working remotely.

In addition, Glass can help maximize evaluator’s multi-tasking behavior (when appropriate).

Lessons Learned – Caveats:

Take time to get to know people before disrupting their norm with this innovation.

Plan to use it over time to allow people to become accustomed to it and drop their company manners.

Respect people’s privacy. Ask for permission to record any behavior.

Do not use it in bathrooms, while driving, or in areas requiring additional sensitivity, e.g. bars, gang gatherings, and funerals.

In the short term, expect the shock factor, concerns about invasion of privacy, and a lot of attention. Over time, as the novelty wears off and they become more common place, Glass will be less obtrusive than a bag of digital cameras, laptops, and Smartphones.

Rad Resources:

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

David Fetterman“Ok, glass.” That’s how you activate Google Glass. I’m David Fetterman and that’s me to the right wearing Google Glass. I’m an empowerment evaluation synergist and consultant, busy father and spouse, and owner of Fetterman & Associates.

Rad Resource – Google Glass: Google Glass is a voice and gesture activated pair of glasses that lets you connect with the world through the internet. You can take a picture, record a video, send a message, listen to music, or make a telephone or video call – all hands free.

Hot Tips – Redefining Communications: Google Glass is not just another expensive (currently about $1500) gadget. It can free us up to do what we do best – think, communicate, facilitate, and, in our case, assess. Here is a brief example.

I said “Ok, Glass,” then “make a call to Kimberly James.” She is a Planning and Evaluation Research Officer I am working with at the W.K. Kellogg Foundation.

Kimberly asked how the evaluation capacity building webinar is coming along. Via Glass, I took a screenshot and mailed it to her so we can discuss it. When a colleague is mentioned, with a few swipes of my finger on the frame, I find a picture on the web, and miraculously remember who we are talking about.

Mid-conversation, Kimberly needed to step away briefly. While on hold, I sent a note to colleagues in Arkansas to ask them to check on the data collection for our tobacco prevention empowerment evaluation.

Kimberly returned to the call and we discussed a recent survey. With a simple request, the display of our results appeared, reminding me what the patterns look like.

Did I mention that I did all of these things while making lunch, picking up my son’s clothes off the floor, letting the dogs out, and emptying the dishwasher?

Later in the day, with a tap on the frame, I confirmed our scope of work with Linh Nguyen, the Vice President of Learning and Impact at the Foundation, while dropping my son off for piano lessons.

Later in the week I plan to use Google Hangout to videoconference with another colleague using Glass. When she connects during a project site visit, she will be able to take pictures and stream video of her walk around the facilities, bringing me closer to the “hum and buzz” of site activities.

Lessons Learned:

Respect people’s privacy – do not wear Google Glass where it is not wanted, will put people off, or will disrupt activities. Do not take pictures without permission. Remove it when you enter a bathroom.

Rad Resources

Hot Tip: Stay tuned for Part II tomorrow when I will cover using Google Glass as an evaluation tool.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Enhanced by Zemanta


My name is Susan Kistler and I am on a crusade to expand our reporting horizons. Earlier this month, we looked at little chocolate reports. Today, let’s consider adding videos to your evaluation reporting toolbox.

aea365_videos_suck_cover

Get Involved: But first, a little incentive for you to share your best alternative reporting ideas. And possibly get a reward for doing it. In the notes to this blog, or via twitter using the hashtag #altreporting, share either (a) your best unique evaluation reporting idea, or (b) a link to a great alternative evaluation report, and in either case note why you love it. I’ll randomly draw one winner from among the commenters/tweeters and send you a copy of “How to Shoot Video That Doesn’t Suck,” a book that can help anyone create video that isn’t embarrassing. Contribute as often as you like, but you will be entered only once in the random drawing on May 1.

Back to our programming. If you are reading this via a medium that does not allow you to view the embedded videos, such as most email, please click back through to the blog now by clicking on the title to the post.

Rad Resource – Unique Reporting Videos: Kate Tinworth, via a post on her always thought-provoking ExposeYourMuseum blog, recently shared three wonderful short video reports made by her audience insights team when she was working at the Denver Museum of Nature and Science. Each uses everyday objects to help visualize evaluation findings in an engaging way.

This video is my favorite of the three. It introduces the evaluators, reports demographics via a stacked bar chart built from jellybeans, and is at once professional and accessible.

Cool Trick: Kate’s team met museum volunteers and staff at the door with small bags of jellybeans that included a cryptic link to the report in order to get people to view the video.

Rad Resource – Unique Reporting Videos: This video from a team in Melbourne, Australia, shares findings from an evaluation of a primary school kitchen gardening program. It introduces the key stakeholders and deepens our understanding of the program without listing its components.

Rad Resource – Unique Reporting Videos: I wrote before on aea365 about getting this mock reporting video made for $5. I can still envision it embedded on an animal shelter’s website, noting how the shelter is using its evaluation findings. My favorite part is that it talks about evaluation use – how things are changing because of the evaluation at a small business.

Rad Resource: Visit the Alternative Reporting – Videos Pinterest Page I’m curating for TheSmarterOne.com for more reporting video examples and commentary.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Road Less TraveledHello fellow evaluators! My name is Ann Price and I am President of Community Evaluation Solutions, near Atlanta, Georgia. A few weeks ago a friend and I spent the weekend in the Georgia Mountains at the Hike Inn, a state park only accessible via a 5 mile “moderate” hike. There is no cell phone, no tv, no internet. It was nice to disconnect and reflect on life and work. This blog about my reflections over the weekend as an external evaluation consultant.

My friend and I have found over the years that even though we work in different areas, our processes and our relationships with clients are quite similar. We both have a penchant for metaphor so we had fun over the weekend applying metaphors to our clients and our work.

The first thing we did was spend ½ hour just trying to find the trail head. I told my friend this was similar to programs not doing the ground work for an evaluation (i.e. failing to design a program logic model or a strategic plan or in our case, having the map but not following it). When all else fails, read the directions….

The hike was a lovely, albeit up and down trek. So the second thing we learned was something my son’s scout leader once said, “Everyone is on their own hike.” We reminded ourselves of that as folks of all ages passed us by (that was a bit discouraging). But the main point is to start on the path. Similarly, you may not have the biggest, most well-funded program. But it is important to start the evaluation journey or you will never “get there.” You do this by building your program’s organizational and evaluation capacity.

Tips and Tricks:
The hike was pretty steep at times, so we had to stop every once in awhile and catch our breath. We kept ourselves motivated by setting goals (Let’s just make it to the next tree! Think benchmarks and indicators). Evaluation work is the same way. It’s important to take a break and look at your data. If you don’t you might miss some pretty awesome sites (or findings). So stop every once in awhile and see where you are. Is your program where it needs to be? If your program is not, make an adjustment. And if you need help, here are a few great resources to guide you on your way.

Rad Resources:
Start with baby steps if you must. There are plenty of free resources out there to help you on your journey:

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Greetings to my fellow #DataNerds! My name is Jordan Slice. I am a Research Specialist at Richland One, an urban school district in Columbia, South Carolina. In addition to being a full-time evaluator, I create handmade pieces for my Etsy shop, resliced.

As a handmade business owner, many of the sales I make are custom orders. People really appreciate when something is tailored to meet their needs. The same is true for evaluation stakeholders: your results are much more likely to be appreciated (and used!) if they answer the questions your stakeholders need to know.

Lesson Learned: Whether I’m making a custom purse (that’s one of my bags to the right) or designing a program evaluation, clear communication is key. For example, if a customer sends me her grandfather’s favorite shirt and requests that I make her a purse using the fabric, it is imperative that we come to a clear agreement about the design of the purse before I start constructing. Similarly, when evaluating a program, it is imperative that you consult with the stakeholders before developing your evaluation if you expect the results to be utilized.

Hot Tip: Keep it simple. While you and I may love geek speak, flooding your stakeholders with evaluation jargon may impair their ability to understand your results. Whether you are talking with stakeholders, constructing a presentation, or writing a report, commit to the mantra that less is more. Once I gather my summary in writing, I use a two step revision process. First, I focus on organizing the content for better flow. Second, I put on my minimalist cap and cut out all the excess fluff (usually repetitive statements or unnecessary detail). Before finalizing any report, always ask a colleague (or stakeholder when appropriate) to proof and provide feedback. I employ the same technique when I am building newsletters (Rad Resource: Mail Chimp – free & user-friendly!) or item listings on Etsy.

Rad Resource: Stephanie Evergreen has some really great posts (like this one!) on her blog with tips for creating better visualizations with your data.

Another Hot Tip: Allow yourself time to focus on something creative (even just a daydream) several times a week. This can give your mind the break it needs to process information and improve your focus. Pursue a new hobby or build on an existing interest. You may be surprised at how this new skill can help you grow as an evaluator.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Older posts >>

Archives

To top