AEA365 | A Tip-a-Day by and for Evaluators

CAT | Uncategorized

Kristin Fields

Kristin Fields

My name is Kristin Fields and I am the editor for AEA’s monthly e-newsletter. I joined the AEA staff about a year ago, and it has been such a pleasure to learn about our many members’ experiences in the field of evaluation.

Our e-newsletter is a chance to share the latest and greatest AEA news, events and educational opportunities with you. It’s also a great place to learn about your fellow members, whether it’s through columns like “The Face of AEA,” which spotlights member experiences, or sharing updates from our many AEA Topical Interest Groups. By getting to know your fellow AEA members, you may find you share a common experience, and down the road they can be a resource to help you solve a challenge. Learning from others is a powerful tool, and with that in mind, I’d like to share a few tips for interviewing. Not a writer? Not a problem. With Evaluation 2018 right around the corner, these tips can also be applied to the many networking opportunities you’ll have between sessions.

Hot Tip: Come prepared with a list of questions.

There’s nothing worse than an unprepared interviewer. When asking someone to take time out of their day for you, make sure you have a list of questions handy. Whenever possible, share questions in advance to ensure a productive conversation. While networking doesn’t necessitate the same procedure, it never hurts to mentally prepare a few basic questions (i.e. “How long have you been in the field? What got you started? What are some of the challenges/successes you’ve faced recently?”)

Hot Tip: Pay attention.

This seems obvious – however, there are so many distractions that can ruin a good conversation. When interviewing someone for a story, it’s important to ask if you can record the conversation. This lets you avoid typing notes as they speak, so you’re fully present and the click-clacks of your keyboard don’t detract from the conversation. While you don’t need to record conversations while networking, similar rules apply: put your device down, maintain eye contact and focus on what the person is saying so the conversation flows naturally.

Hot Tip: Follow up is key!

Once the interview is published, be sure to follow-up with the interviewee and thank them for their time. If applicable, share a link to the article online and encourage them to share it with others, too. In networking situations, don’t forget to get the person’s contact information, and follow-up with a quick note to thank them for their time. This can lead to future conversations and greater connections.

Whether you’re interviewing someone for a blog or making a connection in person, I hope these tips lead to productive conversations and future relationship building.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

Hello, my name is Tania Jarosewich, owner of Censeo Group, a northeast Ohio program evaluation firm that supports educational agencies and nonprofit organizations in evaluating their programmatic efforts. We northeast Ohio evaluators are excited to welcome you to Cleveland for Evaluation 2018.

Check out the Rad Resources below for a list of Cleveland-focused Twitter feeds. Why Twitter? Well, it’s an easy and quick way to learn about upcoming events and a good place to learn about new evaluation results and research findings.

Welcome to Twitter displayed on tablet

Max Pixel Creative Commons Zero – CC0.

It’s easy to post a link and quickly share information with followers, but is Twitter an effective way to share work and build knowledge in the evaluation community? Research about the impact of tweeting on article or report downloads and citations has been equivocal. One study found a correlation between Twitter activity and higher citation rates in the area of ecological research. However, other studies have not found differences in the number of citations based on whether or not the author is on Twitter. Although there might not be a clear correlation between Twitter and citations, there is a benefit of using Twitter to disseminate evaluation or research findings. Twitter users who have a large following and compelling or interesting findings, have a greater chance of sharing findings with an audience wider than colleagues who would read a scientific journal, for example increasing visibility on Google searches. Tweeting about a project or finding, a relatively quick task, could amplify the message and communicate with a wide and varied audience.

Rad Resources: Add these Twitter handles to your feed to get a sense of what is happening in Cleveland and find places to visit during the conference.

Destination Cleveland @TheCLE

#HappyinCLE @happyincle

Cleveland Magazine @ClevelandMag

Cleveland Foundation @CleveFoundation

Downtown Cleveland @DowntownCLE

Cleveland Scene @ClevelandScene

West Side Market @WestSideMarket

Cleveland Metroparks @clevemetroparks

Rad Resource: Jayne Corso, the voice behind the @aeaweb Twitter page has written a number of AEA 365 posts about evaluators who tweet

Hot Tip: use the #Eval18 hashtag to tweet about or follow tweets about the AEA 2018 conference.

We’re looking forward to the fall and the Evaluation 2018 conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

· ·

I’m Cindi Szymanski, Senior Manager of Brand Marketing & Communications for the largest performing arts center in the country outside of New York. On behalf of everyone at Playhouse Square, please allow me to welcome you to our great city!

Playhouse Square has a total of 11 performance spaces and welcomes more than a million guests each year. We present touring Broadway, concerts, comedy, dance and family shows. Playhouse Square is home to six resident companies: Cleveland Ballet, Cleveland Play House, Cleveland State University Department of Theatre and Dance, DANCECleveland, Great Lakes Theater and Tri-C JazzFest.

Five of our theaters were built in the early 1920s: the Allen, Connor Palace, Hanna, KeyBank State and Ohio. They are beautifully restored and if you like, you are welcome to tour them on Saturday, November 3. Tours are free and depart every 15 minutes 10-11:30 a.m. from the KeyBank State Theatre lobby (1519 Euclid Avenue). Wear comfortable walking shoes; the tours last 90 minutes.

Lessons Learned:

Evaluators might be interested to the answers to some of the following questions:

  1. How are the Broadway musicals selected for Cleveland? Executive producer, Gina Vernaci, curates the KeyBank Broadway Series. As a Tony voter, she sees all of the productions in New York and then negotiates with producers to create a well-rounded season for Cleveland audiences.
  2. Who are the audiences and where do they come from? Approximately 10% of the Playhouse Square ticket buyers come from outside a 50-mile radius of Cleveland. This is determined through data collected during the purchase process, where we collect name, address, phone and email. As for other demographics (age, income, gender, etc.), we occasionally distribute surveys but that information is always optional to provide.
  3. How do ticket prices in Cleveland compare to New York/Broadway? There is a lot of variation in NY ticket pricing, but ticket prices are lower in Cleveland.

Hot Tip:

  • Photo Op: In the heart of our neighborhood is a great photo opportunity with the GE Chandelier, the world’s largest outdoor chandelier, suspended right over Euclid Avenue and bedazzled with 4200 crystals. You’ll find a number of bars and restaurants within walking distance and there’s something to fit every taste – Italian, Mexican, Korean, German and more.

Rad Resource:

Enjoy your visit to Cleveland!

 

We’re looking forward to the fall and the Evaluation 2018 conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

Hello!  I’m Clara Pelfrey, a native Clevelander, a board member of the Ohio Program Evaluators Group and a member of the Local Arrangements Working Group for Evaluation 2018. I’d like to welcome you to a week of blogs about evaluation in Cleveland as well as tell you about my fair city and the things you can see and do. For discounts, visit the Evaluation 2018 website.

Cleveland skyline at night

Image Credit: Carlos Javier via Flickr

Rad Resources:

Arts & Culture.  The Cleveland Museum of Art is free to visit and has one of the world’s best collections. While you’re there you can stroll around the Wade Lagoon and admire Rodin’s “The Thinker”. The Cleveland Museum of Natural History has exhibits from dinosaurs to diamonds and don’t miss the outdoor Ralph Perkins II Wildlife Center & Woods Garden, a wonderful outdoor exhibit with live native animals that can climb through catwalks above your head. Cleveland’s Museum of Contemporary Art (MOCA) is free on the first Saturday of the month. The Rock and Roll Hall of Fame (aka the “Rock Hall”) has Elvis Presley’s motorcycle and Michael Jackson’s white glove and you can be a pinball wizard at the Rock & Pinball interactive exhibit. If you’re bringing your children, you’ll love the Great Lakes Science Center with its myriad of interactive exhibits, polymer fun house and IMAX theater. Theatergoers, be sure to catch one of the many shows at Playhouse Square, discounted for AEA. For classical music lovers, visit Severance Hall to experience the world-renowned Cleveland Orchestra for “Gerstein plays Rachmaninoff”.

Nature and the Outdoors. Immerse yourself in the Madagascar or Costa Rica habitats at the Cleveland Botanical Gardens.  Lakeview Cemetery is an outdoor museum, where you can visit President James A. Garfield’s Memorial and see breathtaking Tiffany glass in the Wade Memorial Chapel. For hiking or biking, be sure to visit the Cuyahoga Valley National Park. The Cuyahoga Valley Scenic Railroad, with its vintage engine and cars, passes old lumber and grain mills, historic villages, and art exhibits as it chugs through the valley along the Towpath Trail, a former stretch of the Ohio and Erie Canal. Don’t forget the awesome Cleveland Metroparks Zoo.

Totally unique.  The Victorian-era Old Arcade dates from 1890 and was the first indoor shopping center in America. The West Side Market is one of the oldest continuously operating food markets in the country. Visit the house where “A Christmas Story” was filmed in Tremont.

Food, drink, fun. The Jack Casino is walking distance from the convention. There are many restaurants on the east bank of the Cuyahoga river (aka “The Flats”), Downtown on pedestrian-friendly East 4th street or on West 3rd street.  Other places with lots of character and many restaurants include the Coventry Village area in Cleveland Heights or Ohio City and Tremont, which are on the west bank of the  Cuyahoga River, where Irish immigrants settled after helping build the Ohio & Erie Canal. If you’re looking for microbreweries, try the Great Lakes Brewing Company, the first microbrewery in Ohio.

 

We’re looking forward to the fall and the Evaluation 2018 conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

Hi, I’m Sara Vaca, independent evaluator working in international development evaluation, helping Sheila curate this blog and being a Saturday contributor, and I want to share with you a short reflection on something that is bugging me lately.

Conducting an evaluation is a process: a rich, participatory, multi-partner, multidimensional process, that has to be later condensed, reflected, gathered in a report (a product), that is unidimensional, static and written by the evaluation leader or team. I have realized some time ago that those two parts of the assignment, equally important, are soooo different! And this represents a challenge in my practice, as I wonder: how can I make the report better represent the process?

Cool Trick: First, of course, try to make the evaluation process as technically good and adapted to the commissioners’ needs as possible. If the process is not good (see this related post), nobody is probably going to care about the report. In general, I enjoy conducing the evaluation process so much, and the feedback about the process is usually good or very good.

Ok, but what about the report?

That is when the second, less rewarding in my experience phase starts.

No matter what you get as findings, internal validity of the process has so far granted me acceptance during the debriefing and validation phase. However, the evaluation users are waiting to see it in a document, clearly stated and articulated, to be able to read it and assimilate what you saw through the process. And at that point, I notice a slight change of attitudes in them, and in consequence in me, as I have to adapt:

And I totally understand: the process is a soft activity and not easy to see the traces of it, but the report is the hard activity, and what it says, stays there forever… Still I find it very interesting how both parts of the same assignment differ in so many things.

Hot Tip: The only thing I’m doing so far is to make the materials in the debriefing presentation very consistent with the evaluation report. How I do that? I create summaries and visuals for my Powerpoint presentation (that I share with them when I leave the country) that will later be quite similarly reproduced in the draft.

However, looking forward to your ideas and tips… How do you reconcile these two equally important parts of evaluation? Thanks!

Sep/18

3

Are you really coaching? by Betsy Block

Hi! I’m Betsy Block, coach, consultant and jill-of-all-trades.  I added coaching to my capacity building work as I trained with two thought pioneers– Coaching Training Institute and CRR Global.

Evaluators have started asking critical, challenging questions: how do we build our client’s capacity? How do we incorporate equity into our own practices? How do we guarantee lasting, meaningful use of evaluation findings?  While independent, evaluation still has a critical role, more often evaluators are seeking alternate ways to increase integration of findings into practice.  In that search, evaluators are increasingly integrating non-traditional tools focused on the interpersonal side as innovative capacity building approaches.

Hot Tip:

Consulting, coaching and mentoring are all approaches we use in with clients to help drive adoption. Try focusing on which approach you are leaning into with your client and the implications of that.

Consulting, mentoring, coaching

Coaches are awesome.

Coaches encourage you, believe in you, and help you achieve your best!  The increase in evaluators offering coaching is a great trend, maybe in part because it feels more approachable than traditional consulting models.    Many evaluators received accredited coach training or hold certifications from the International Coach Federation, the largest accrediting body for coaching.  Many instinctively coach because it is just their way of being.  However, some offer coaching but deviate from evidence-based practices that trained and/or credentialed coaches use – and while these consultants can get positive results, they risk missing the mark in terms of equity and more impactful capacity building.

So how do you do more coaching? 

Lessons Learned:

  • Three simple words can make coaching happen: tell me more.
  • Hold clients as creative, resourceful and whole: prize their expertise above yours. It’s harder than we think. (Let’s get real, we are experts with cool tools!) Last summer, a client began our meeting by asking me to tell him what to do.  I shelved my expertise and told him he had all the knowledge to lead the project.  I asked him simply “What are you wanting?” and kept asking curious, short questions. He had already conceived a thoughtful, purposeful evaluation framework.
  • Coach the system. You are often coaching more than a person, and more than a group of people. Have you ever walked into your client’s office and felt that feeling in the air, like it had its own presence?  That’s the system – you have to reveal it to your client so they can figure out what they want to do about it.   Don’t judge or characterize, just notice it and give them a safe space to talk about it.  “I feel some kind of energy in this room in this room? What about you?”

Rad Resources:

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hi, we are Sy Islam, Michael Chetta, and Andrzej Kozikowski from Talent Metrics, a data driven consulting firm. We’re writing about using text analysis to identify new areas of improvement when conducting an evaluation. Evaluation research often uses surveys or other forms of data collection that limit potential outcomes of the evaluation. When designing a survey, you may not cover all the areas of potential improvement using fixed response scale items. Text analysis is a phenomenological approach that allows research participants to provide more detailed responses, typically to open-ended questions. Rather than constraining responses that you think participants will give, text analysis and open-ended questions allow research participants to provide their own ideas and insights into the workplace. Qualitative approaches can be used to gain insight on quantitative results. Text analysis is an efficient way to analyze large amounts of qualitative data rather than traditional methods that require more time and resources.

Hot Tip: Use open-ended questions, social media comments, and text analysis to look for areas of improvement and areas of weakness in an intervention or program.

When evaluating a program or site, open-ended items should be included in any evaluative survey. Another great source for open ended comments would be social media posts and pages. Every major business or service has a Facebook, Yelp, or Google profile. These resources can provide rich and surprising data.

We conducted an evaluation of urgent care centers to identify what language drove positive as well as negative ratings.  Organizing comments from Yelp into positive (4 or 5) and negative (3 and below), we identified language that was used most often in those ratings.  We had hypothesized that different language drove urgent care ratings on social media, and that they fell into three broad categories: 1) Facility, 2) Care Delivery and 3) Staff. We were surprised that patient reviews, both positive and negative, were driven by the same types of experiences.

Top words found in positive and negative reviews

Rad Resource: We used Tropes 8.4 to analyze the text from the social media reviews. Tropes is an open source software used in academic and applied research. It can also provide a graph of actors like the one below that illustrates the relationships between review quality and the language that is used.

Graph of Actors: Relationship of "negative reviews" to patient evaluation terms of interest

Lessons Learned: Evaluation of businesses, nonprofits, and their associated programs are often conducted using surveys or other forms of quantitative data collection. These methods often include the evaluators’ own assumptions about what is important in driving key outcomes. By using qualitative data, such as social media comments, new and unexpected discoveries can be made. Remember to listen to program participants or those who utilize your services. They are the real subject matter experts and can help you as you evaluate.

Rad Resources:

Two  text analysis articles we wrote that go into further detail regarding our projects.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello! I’m Sheila B Robinson, aea365’s Lead Curator and sometimes Saturday contributor and I’m getting excited about my presentations at Evaluation 2018! I’m thinking about what content to share, what I want my participants to know, how I will present, what my slides will look like…so much to consider! That’s why I love p2i!

Rad Resource: AEA’s Potent Presentations Initiative (p2i) exists for the explicit purpose of helping evaluators improve their presentation skills, both at conferences and in individual evaluation practice. Potent Presenters think about three key components of a compelling presentation: Message, Design, and Delivery, and our free resources are largely organized around these areas. The p2i website features a p2i Presentation Tools & Guidelines page with downloadable checklists and worksheets, along with webinars and slides, and did I mention that they are all free?

Lesson Learned: Study successful presenters. P2i got its start back in 2012 with Stephanie Evergreen, data visualization and presentations expert, and has continued to grow over the years, with several people contributing to the effort, including me! The three key components and inspiration for much of the original content and came from a study of a dozen well-known AEA presenters – the “Dynamic Dozen” – who had the highest feedback scores from pre-conference professional development, AEA Summer Institute workshops, and Coffee Break Webinars.

Get a pulse from current presenters. As current p2i coordinator, I’m always on the lookout for ideas for new content. In a recent effort to catalyze ideas I asked evaluators to participate in a brief, informal survey about presenting to help inform our future work. The survey asked about people’s past and future presentation work and what might help them improve their practice. I shared the link to “You, the Presenter: What Would Help You Up Your Game?” in an aea365 post and on EvalTalk, and 190 people responded! The results are quite informative, and I shared them over several months in the AEA monthly newsletter in my p2i column.

Rad Resources: 

Are you presenting at #Eval18 or another conference or event? It’s a great time to check out AEA’s Potent Presentations Initiative and download free resources to supercharge your next presentation!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hi my name is Jayne Corso and I am the Community Manager for AEA.

Planning your social media calendar in advance can keep you from scrabbling to find rich content to share. That’s why I recommend a social media content calendar! Through a social media content calendar, you can schedule posts days, weeks, and months in advance.

Hot Tip: Identify content that resonates with your followers

Before you go too far down the path of creating a social media content calendar, first identify the type of posts that receive the most engagement from your followers. You can do this by looking at past posts and seeing what topics received the most comments, shares, and likes. You can also look back to see if your followers engaged more with links, photos, or videos. This will help you define your content.

Hot Tip: Decide how often you want to post

How active are you going to be on your social media platforms? Do you have the bandwidth to post multiple times a day, or will multiple times a week make more sense? When choosing this timeframe, be realistic with your available time and the amount of content you have to share. Here is a listing of the minimum posting schedules for social media platforms from Constant Contact.

Facebook: 3x per week

Twitter: 5x per week

LinkedIn: 2x per week

Hot Tip: Map out important dates

If you know you have events coming up or important deadlines, place these dates on your calendar. This will help remind you that a webinar, conference, or submission deadline is approaching and more communication efforts need to be made for this topic. You can also place fun reminders on your calendar such as holidays or anniversaries.

Hot Tip: Find outside content to share

It’s great to share your own content, but that’s not very social. Using social media should be a social experience where users are sharing content. Identify blogs, website, or other social media pages that have content relevant to your followers. Set aside days on your social calendar to share postings from these sites. I find BetterEvaluation, @WashEval, and conversations using #DataViz to be great resources for content.

Now you can create your own social media content calendar. I use a simple excel spreadsheet, but there are many ways you can create your calendar, including online platforms such as Hootsuite and Sprout Social. Get busy posting!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Greetings, colleagues.  I’m Stephanie Cabell and a Program Analyst at the U.S. Department of State.  I work as an evaluation advisor and performance management specialist.  Together with colleagues at our partner agency—the U.S. Agency for International Development—we work to identify effective approaches and methods to evaluate the two main pillars of our work–diplomacy and development.

Western nations have long approaches to illuminate the nature of diplomacy and diplomatic activities, and to establish effective evaluation systems in order to measure and evaluate diplomatic successes.  The U.S. Department of State undertook a study to gather insights to help shape its efforts to construct an evaluation process for assessing the effectiveness of diplomatic programs.  Key questions we asked ourselves:

  • What kinds of diplomatic programs/initiatives would lend themselves most easily to evaluation?
  • What criteria could be used when deciding to evaluate?
  • What approaches should be used to make evaluations of “diplomacy” more meaningful and useful to government leaders?

Over the course of several rounds of structured interviews with current and retired diplomats and other subject matter experts, here’s what we learned.

Hot Tips and Rad Resources:

Challenges to Evaluating Diplomacy: It’s no surprise that diplomatic activities that require long-term negotiations, global and regional coalition building are hardest to measure.  These include:

Hot Tips:

  • Diplomacy is not easily quantifiable: It is often a fluid, “amorphous,” non-linear process.
  • Issues of attribution: The U.S., as is true for all nations, is not the only actor in the foreign policy arena.
  • Diplomats tackle complex issues that cannot be easily understood and captured.
  • Criteria that may be used to select diplomatic efforts for evaluation purposes include technical feasibility, the cost-benefit equation, political sensitivity, and urgency of the information.

Rad Resources:

Additional Rad Resources: For those wanting a deeper dive into this area, related articles and resources on evaluating “diplomacy” include the following:

 

The American Evaluation Association is celebrating Washington Evaluators (WE) Affiliate Week. The contributions all this week to aea365 come from WE Affiliate members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Older posts >>

Archives

To top