AEA365 | A Tip-a-Day by and for Evaluators

CAT | Teaching of Evaluation

I am Susan Kistler, the American Evaluation Association’s Executive Director and aea365’s regular Saturday contributor.

Guess what just moved into the #1 spot for the most read aea365 article of all time? The April 6, 2013 post announcing the collaboration with BetterEvaluation to produce a series of publicly available Coffee Break Webinars! We’re well under way with the webinar series focusing on the RainbowFramework for planning, managing, and implementing an evaluation, and I want to share an update:

Hot Tip – Sign Up to Attend the Remaining Offerings Live: There are still six more webinars to go in the series, to be offered every Tuesday and Thursday through until the end of May. If you attend live, you have an opportunity to raise questions at the end, and to hear directly from the presenters. Register for the remaining free webinars here – note that you must register for each one separately.

Hot Tip – Subscribe to AEA’s YouTube Channel and Check Out the First Webinar in the Series: The recording of the first webinar in the series is already available online. Go to http://www.youtube.com/user/AmEvalAssn and click the Subscribe button just below the header to receive notices as each subsequent recording is posted.

Hot Tip – We’re having each of the videos in the series professional transcribed: The transcription is useful in at least four ways: (1) it enables reading the video content for those who may have hearing limitations, (2) it allows viewers to read along in case the video’s audio is not perfect or the viewer is more comfortable with written rather than spoken English, (3) it allows for easily extracting references and written quotes from the video, and (4) it serves as the basis for improved translation to other languages.

Cool Trick – Automated Translation Is Available and Human Translations Are Coming: You can select “Translate Captions” (see the screenshot below) and Google will create a machine translation of the captions and transcript. Working from a professional transcription improves the quality of the machine translation, but the quality is still quite variable. I am excited to announce that we are working with a team of volunteers to translate this video series into multiple languages.

Lessons Learned (Coming Soon): I’ll share a note once the human translations are available. And, in a future post I will also cover (a) how to contract for transcriptions, (b) how to work with colleagues to create video translations (it’s easier than you think), and (c) how to upload/incorporate transcriptions and translations into your videos since I know that increasingly we’re seeing evaluators sharing video on YouTube.

Hot Tip – Make the Most of YouTube:

YouTubeHowTo

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I am Susan Kistler, the American Evaluation Association’s Executive Director and aea365’s regular Saturday contributor.

Hot Tip – Collaboration with BetterEvaluation: Today, we’re excited to announce a partnership with BetterEvaluation. BetterEvaluation is an international collaboration to improve evaluation practice and theory by sharing information about options (methods or tools) and approaches.

Rad Resource – BetterEvaluation Rainbow Framework: BetterEvaluation has produced a framework that organizes over 200 evaluation options into 7 clusters of evaluation tasks that can help you to plan and manage an evaluation. On their website at http://betterevaluation.org/ you’ll find the framework itself as well as extensive explanation and examples in support of the framework tasks. This resource brings together contributions from evaluators working on the ground in various contexts around the world.

 

Hot Tip – Series of eight free short-form webinar trainings: BetterEvaluation and the American Evaluation Association are teaming up to bring to you a series of eight Coffee Break Webinars in May. This series is open to the public (please help us to spread the word), registration is free, and the speakers represent deep expertise applicable in both domestic and international contexts. The series of eight webinars walks you through the components of the Rainbow Framework and will include takeaways immediately applicable to your practice:

  1. Overview of Rainbow Framework for Evaluation – Irene Guijt
  2. Define What Is To Be Evaluated – Simon Hearn
  3. Frame the Boundaries of the Evaluation – Patricia Rogers
  4. Describe Activities, Results and Context – Irene Guijt
  5. Understand Causes of Outcomes and Impacts – Jane Davidson
  6. Synthesise Data from One or More Evaluations – Patricia Rogers
  7. Report and Support Use of Findings – Simon Hearn
  8. Manage an Evaluation – Kerry Bruce

Pre-registration is required and you can register for as many as you would like here: http://comm.eval.org/coffee_break_webinars/CoffeeBreak/BetterEvalSeries

Rad Resource: In 2012, AEA co-hosted a coffee break webinar series with Catholic Relief Services, USAID, and the American Red Cross. This collaboration resulted in a series of four public coffee break offerings, and anyone – AEA member or not – may view the recordings from all four online here. Included in this set are:

  • Monitoring and Evaluation Planning for Projects/Programs
  • Evaluation Jitters Part I and Part II
  • Simple Measurement of Indicators and Learning for Evidence-Based Reporting

We hope to work further with this wonderful team in the future, and currently Scott Chaplowe who helped to spearhead this series is offering an AEA eStudy.

Rad Resource: AEA hosts a weekly Coffee Break Webinar series for members only. AEA members are welcome to attend any of the Thursday afternoon offerings live throughout the year (see list of upcoming offerings here), or to access recordings of over 100 coffee break webinars via the webinars archive (see the public list of what’s in the archive here). If you aren’t currently a member, we encourage you to join!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello, we’re Dominica McBride and Leah Neubauer, members of the AEA Public Statement on Cultural Competence Dissemination Working Group. At AEA 2012, we participated in the brownbag session Critically Reflecting and Thinking about Cultural Competence in Your Evaluation Practice. We’d like to share some of the great insights generated from this session.

In this session, we learned about a nursing student who, influenced by Western Medicine cultural ideologies of time- and task-orientation and protocol, misdiagnosed a patient due to a lack of cultural sensitivity and critical self-awareness. We also learned about a comprehensive model of self-reflection from Hazel Symonette that encompasses internal, external/other, space, and time. Below are several lessons learned:

Lessons Learned

Our sense of time can affect how we do our evaluations and the use and impact of our evaluations. In the case study of the nursing student, she quickly ran through her nursing duties, completing an assessment and diagnosis in under 5 minutes. This rush caused a missed opportunity to gain more insight and the patient was misdiagnosed. We, as evaluators, can do this same thing. If operating in a mental frame of “getting stuff done” and “getting it done fast”, we miss opportunities to collect valuable data and gain additional insight. We also miss the opportunity to be more self-reflective. Without this introspection, we may, like the nursing student, “misdiagnose” or misinterpret the situation and data.

We too often assume that we know. Even in the midst of data collection and trying to be objective, we can often assume that we know something or have an accurate interpretation when we actually do not. In Love’s Executioner, Irvin Yolam, a reputable psychiatrist, talks about how communication is filtered through our experiences, biases, feelings, etc. We never fully accurately know what the other is communicating. However, with self-reflection and dialogue, we can check to see if we’re interpreting correctly. We create spaces to gain other perspectives accurately, thus enriching our interpretations and leading to accurate and comprehensive work.

We strive for perfection – is perfection possible?  In Becoming a Critically Reflective Teacher, Dr. Stephen D. Brookfield discusses teaching culture that strives for the ‘perfect 10’ in their teaching. A critically-reflective state of practice acknowledges a constant state of knowing and learning more.

One lesson harkens back to a lesson from SpidermanWith great power comes great responsibility. As evaluators, our judgments are seen as mattering more than some others. We have a special responsibility to ensure that findings and conclusions are inclusive, public, and shared.

Rad Resources

Hazel Symonette developed a comprehensive model for self-reflection we can easily incorporate into evaluation and daily life.

Stephen D. Brookfield has written extensively on the role of critical self-reflection for adult educators.

This week, we’re diving into issues of Cultural Competence in Evaluation with AEA’s Statement on Cultural Competence in Evaluation Dissemination Working Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Jean King and Laura Pejsa, Minnesota Evaluation Studies Institute (MESI), here, with broad smiles on our faces. We are the proud coaches who are wrapping up this week of posts written by our creative student consultants about ways to evaluate a conference (using exit surveys of Ignite sessions, network visualizing, Twitter, and video clips).  Progressive educators long ago documented the value of experiential learning–“learning by doing”–and our experiences during this year’s AEA conference again provide support for the idea as a means of teaching evaluation. Thoughts about how to use a conference setting to engage evaluation students follow.

Hot Tips:

  • Create an evaluation team. Our experience at MESI confirms the value of having students collaborate on projects. Not only do they learn how to do evaluation tasks, but they also learn how to collaborate, an important skill set for evaluators, regardless of their eventual practice.
  • Encourage innovation. Our charge was to think broadly about conference evaluation. At our first meeting, students brainstormed many possible ways to collect data at the conference, no holds barred, the more creative, the better.  As we sought to be “cutting edge,” technology played a role in each of the four methods selected.
  • Make assignments and hold people accountable. Social psychology explains the merit of interdependence when working on a task. We divided into four work groups, each of which operated independently, touching base with us as needed. Work groups knew they were responsible for putting their process together and being ready at the conference. As coaches, we did not micromanage.
  • Make the process fun. University of Minnesota students take evaluation seriously, but their conference evaluation work generated a great deal of laughter. In one sense it was high-stakes evaluation work (we knew people would use the results), but without the pressure of a full-scale program evaluation.

Lessons Learned:

  • Students can learn the evaluation process by collecting data at a conference or other event. Unlike programs, short-term events offer an evaluation venue with multiple data-collection opportunities and fewer complexities than a full-scale educational or social program.
  • A week-long conference offers numerous opportunities to engage in creative data collection. It is a comparatively low-stakes operation since most conference organizers opt for the traditional post-conference “happiness” survey, and any data gathered systematically will likely be of value.
  • Innovative data collection can generate conversation at an evaluation conference.  Many people interacted with the students as they collected data. Most were willing to engage in the process.
  • Minnesota evaluation students really are above average. Garrison Keillor made this observation about Minnesota’s children in general, but this work provided additional positive evidence.

We’re learning all this week from the University of Minnesota Innovative Evaluation Team from Evaluation 2012. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

No tags

Hi, we’re Anne Schwalbe, Keith Miller, and Michelle Gensinger, graduate students in the Evaluation Studies Program at the University of Minnesota.

We chose Twitter as a platform for exploring creative evaluation methods at the AEA Annual Conference. These are the questions we hope to answer with our analysis:

  • How was Twitter used during the conference?
  • Which sessions at the conference generated Tweets?
  • What did Twitter use look like during the conference (descriptive analytics)?

We had a lot to learn about Twitter and conference evaluation. We hope that our experience helps you decide whether Twitter can contribute to your evaluation practice.

Hot TipBe Prepared to Promote Twitter Use: Not all AEA attendees use Twitter. We wanted to increase twitter use for this project.

  • Fun incentives can promote use without making anyone feel pressured.
  • Have instructions about signing up for Twitter readily available.
  • Demonstrate how Twitter can be used. A projector at our information table displayed AEA’s @aeaweb Twitter feed.

Hot TipKnow Your Hashtags (#): Hashtags are essential to analyzing data. They “tag” all the Tweets related to a topic.

  • Hashtags may already exist for an event, BUT you may need to create and promote them. AEA established #eval12 for the conference. We introduced #evalAHA to track learning moments inspired by the conference.
  • A trusted twitter voice, @aeaweb, spread the word about our hashtag.

Hot TipKnow the Limitations of Twitter Data

  • The sample of individuals Tweeting was small and self-selected.
  • Tweets are public.
  • The format (140 characters) is restrictive.
  • Hashtags don’t always work perfectly. Tweets labeled #eval2012 or #aea12 didn’t show up in our initial searches.

Hot TipHave an Analysis Plan

  • Know your time frame and pull data as soon as possible. It gets tricky to grab Twitter data more than a week old.
  • Manual coding of Tweets is time intensive; we’ve spent over 25 hours coding 1,500 Tweets to explore Twitter use at the conference.
  • Free, quick, and easy online tools provide simple descriptive analytics about a hashtag or Twitter account.

Rad Resources

  • Topsy.com: an online tool that instantly displays a summary of activity, top Tweets, and other fun stuff from any hashtag.
  • NCapture: an NVivo 10 add-on that captures data from social media websites and does some auto-coding.
  • Visual.ly: easy infographics templates for hashtags or twitter users. Here’s one for #evalAHA

We’re learning all this week from the University of Minnesota Innovative Evaluation Team from Evaluation 2012. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

Greetings from Jean King and Laura Pejsa from the Minnesota Evaluation Studies Institute (MESI) at the University of Minnesota. This week we will be introducing you to a crop of graduate student evaluators who (we think) made quite a splash at the AEA conference last month.  If you attended, you may have seen one or more of them filming video interviews, conducting on-the-spot i-Pad surveys, tweeting “Aha” moments, or helping participants identify favorite sessions on the giant TIG visualization. If you were not with us at the conference this year, today’s post will give you some background on this project.

It all started with the local arrangements committee for the AEA conference; the committee wanted to add some sparks of evaluation throughout the week and document experiences not captured on the standard after-conference survey. We created a one-credit, special course at the University of Minnesota titled Creative methods for training and event evaluation, and invited students to join us for a grand experiment. The course and the conference activities would be developed based on the interests and ideas of the students in it.

At our first class meeting, we introduced the students to the goals and history of the conference, provided a place (and food) to come together, and gave them the following loose guidelines:

  • to both pilot and model creative ways of documenting conference experiences;
  • to provide some real-time feedback;
  • to make the evaluation process fun/engaging for conference participants;
  • to explore the potential of emerging technologies;
  • to provide meaningful, usable data to AEA;
  • and to make sure they still had time to attend and enjoy the conference themselves.

Hot Tips

  • You don’t have to look much further than your own back yard for meaningful evaluation experiences for students. Instead of simulating or creating projects, check out the events that may already be happening where a little extra evaluation will go a long way.
  • When it comes to creative methods and technology, students can expand our thinking. Give them an opportunity with relatively low stakes, and see the connections they make between the ways they have learned to use things like social media and the evaluation problem.

This week we will be presenting you with more hot tips, cool tricks, rad resources, and lessons learned from this intrepid group of conference evaluators. Days 2-5 of this week will be written by our four student teams: Survey, Video, Network Visualization, and Twitter. We will wrap up the week with a post summarizing what we learned as instructors that may help others in designing meaningful, real-world evaluation experiences for novice evaluators.

We’re learning all this week from the University of Minnesota Innovative Evaluation Team from Evaluation 2012. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

My name is Susan Kistler. I am the American Evaluation Association’s Executive and aea365’s regular Saturday contributor. We’ve been writing a lot lately about AEA’s upcoming conference, Evaluation 2012. Today, I wanted to provide an update on our fourth quarter eStudy workshops – three foundational offerings for filling your evaluation toolbox, even if you can’t join us next month in Minneapolis:

Rad Resource – Introduction to Evaluation with Tom Chapel, Chief Evaluation Office at the Centers for Disease Control and Prevention (CDC)
October 9, 11, 16, & 18, 11:30 AM to 1:30 PM Eastern Time.
Description: This workshop will provide an overview of program evaluation for those with some, but not extensive, prior background in program evaluation. The session will be organized around CDC’s six-step Framework for Program Evaluation in Public Health as well as the four sets of evaluation standards from the Joint Commission on Evaluation Standards. The six steps constitute a comprehensive approach to evaluation. While its origins are in the public health sector, the Framework approach can guide any evaluation. The course will touch on all six steps, but particular emphasis will be put on the early steps, including identification and engagement of stakeholders, creation of logic models, and selecting/focusing evaluation questions.

Last day to register is Tuesday October 2

Rad Resource – Monitoring and Evaluation (M&E) Planning for Programs/Projects with Scott Chaplowe, a Senior M&E Officer at the International Federation of Red Cross and Red Crescent Societies
November 13 & 15, 12:00 PM to 1:30 PM Eastern Time.
Description: This workshop will introduce six key planning steps for a successful monitoring and evaluation system: 1) Identify the purpose and scope of the M&E system; 2) Plan for data collection and management; 3) Plan for data analysis; 4) Plan for information reporting and utilization; 5) Plan for M&E human resources and capacity building; 6) Prepare the M&E budget. The 6-step approach has been designed to guide international programming at both the community and national levels, and is also appropriate for domestic (US) programs and projects. Particular emphasis will be given to planning for data collection and management using an example M&E planning table for indicators and assumptions.

Last day to register is Wednesday November 7

Rad Resource – Creating Surveys to Measure Performance and Assess Needs with Michelle Kobayashi, co-author of Citizen Surveys: a comprehensive guide to making them matter
December 4 & 6, 3:00 PM to 4:30 PM Eastern Time.
Description: Surveys for program evaluation, performance measurement, or needs assessment can provide excellent information for evaluators. However, developing effective surveys requires an eye both to unbiased question design as well as to how the results of the survey will be used. Neglecting these two aspects impacts the success of the survey. This eStudy course will use lecture and homework assignments to review guidelines for survey development. We will use two national surveys, one used for measuring the performance of local governments and the other to assess the needs of older adults, to inform the creation of our own survey instruments.

Last day to register is Tuesday November 27

Hot Tip – Learn more and access the registration page at http://comm.eval.org/coffee_break_webinars/eStudy/

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I’m Susan Kistler, the American Evaluation Association’s Executive Director, and aea365 regular Saturday contributor. Earlier this year, I announced that AEA would be hosting a public series of our normally members-only coffee break webinars. The four video series, held in July, focused on different aspects of International Monitoring and Evaluation.

First, thanks to everyone who made the series possible.  The series co-sponsors are the American Evaluation Association, Catholic Relief Services, American Red Cross/Red Crescent, the United States Agency for International Development, and AEA’s International and Cross Cultural Topical Interest Group (TIG), our presenters were Alice Willard, Scott Chaplowe, Guy Sharrock, and Susan Hahn, and our ever-intrepid facilitator was Stephanie Evergreen.

Hot Tip: AEA is honored not only to have been the host for this series, but to be able to make the full series available publicly, and – a first for AEA thanks in part to our wonderful partners – we’re ensuring that you can not only watch the videos, but embed them on other sites (roll over the video and a ‘share’ icon appears in the upper right hand corner). If you find the content valuable, please share! We ask only that you don’t edit the recordings, so that credit is given where credit is due to our presenters and co-sponsors.

Rad Resources: Here are the four videos from the international M&E series. If you are receiving this post via email, you may need to click back through to the site to view the videos themselves.

Monitoring & Evaluation Planning for Projects/Programs with Scott Chaplowe hosted by the American Evaluation Association
Download a pdf of the PowerPoint slidedeck from AEA’s public eLibrary here (scroll down for the attachment)

Evaluation Jitters Part One: Planning for an Evaluation with Alice Willard hosted by the American Evaluation Association
Download the PowerPoint slidedeck from AEA’s public eLibrary here (scroll down for the attachment)

Evaluation Jitters Part Two: Managing an Evaluation with Alice Willard hosted by the American Evaluation Association
Download the PowerPoint slidedeck and M&E Module on “Managing and Implementing an Evaluation” from AEA’s public eLibrary here (scroll down for the attachments)

SMILER: Simple Measurement of Indicators & Learning for Evidence-based Reporting with Guy Sharrock and Susan Hahn hosted by the American Evaluation Association
Download a PDF of the PowerPoint slidedeck from AEA’s public eLibrary here (scroll to the bottom for the attachment)

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Download the PowerPoint slidedeck from AEA’s public eLibrary here (scroll down for the attachment)

No tags

Hello, I am Maxine Gilling, Research Associate for Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP). I recently completed my dissertation entitled How Politics, Economics, and Technology Influence Evaluation Requirements for Federally Funded Projects: A Historical Study of the Elementary and Secondary Education Act from 1965 to 2005. In this study, I examined the interaction of national political, economic, and technological factors as they influenced the concurrent evolution of federally mandated evaluation requirements.

Lessons Learned:

  • Program evaluation does not take place in a vacuum. The field and profession of program evaluation has grown and expanded over the last four decades and eight administrations due to political, economic, and technological factors.
  • Legislation drives evaluation policy. The Elementary and Secondary Education Act (ESEA) of 1965 established policies to provide “financial assistance to local educational agencies serving areas with concentrations of children from low-income families to expand and improve their educational program” (Public Law 89-10—Apr. 11, 1965). This legislation also had another consequence: it helped drive the establishment of educational program evaluation and the field of evaluation as a profession.
  • Economics influences evaluation policy and practice. For instance in the 1980’s evaluation took a downturn due to the stringent economic policies. Program evaluators resorted to lessons learned through writing journals and books.
  • Technology influences evaluation policy and practice. The rapid emergence of new technologies all contributed to changing goals, standards, and methods and values underlying program evaluation.

Resources:

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · · · · · · ·

I’m Susan Kistler, the American Evaluation Association’s Executive Director and aea365 Saturday contributor. Today I’m writing from Atlanta and the AEA/CDC Summer Evaluation Institute. I wanted to share a bit more about a great resource originally mentioned by Susan Wolfe in her August 2010 post.

Rad Resource – The Community Toolbox: The Community Toolbox provided practical “step-by-step guidance in community building skills.” The Toolbox covers everything from leadership building and group facilitation to developing a strategic plan and organizing for advocacy. Want to review evaluation guidance understandable by a range of stakeholders? Check out the section on Evaluating Community Programs and Initiatives, containing four chapters:

Each chapter is further broken down into multiple sections and each section contains concrete plans, tools and – a unique feature I haven’t seen elsewhere – a PowerPoint summarizing the key points and ready to use to support training and orientation.

Clipped from: ctb.ku.edu (share this clip)

The above represents my own opinions and not necessarily those of the American Evaluation Association. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Older posts >>

Archives

To top