AEA365 | A Tip-a-Day by and for Evaluators

CAT | Evaluation Managers and Supervisors

I’m Sue Griffey; I lead the Evaluation Center at Social & Scientific Systems, Inc. in Silver Spring MD. I mentor outside of work for professionals in evaluation and public health in both formal programs (Cherie Blair. Foundation (CBF), APHA-SA National Mentoring Program, Aspire, Foundation, Rollins School of Public Health Annual Mentoring Program) and through individual connections.

I have noticed over the past few years, as my mentoring work has increased, that my ability to assess mentoring readiness is critical to the success of the mentoring relationship.

Hot Tip: Mentoring is a volunteer acting. Don’t just assume that the Mentor-Mentee pairing results in both being ready. It may appear as Mr+/Me+ (as in the table below) but the pairing may actually be in a discordant cell.

aea365_Griffey1

aea365_griffey2Mentee isn’t ready: The mentee may not realize she isn’t ready for mentoring; you as the mentor may need to help her see that. A mentee may identify needing mentoring when it really isn’t what she needs. As the mentor, develop and apply metrics for readiness as you would in an evaluation.

Hot Tip: I have a 3-email rule. If I have to track down the mentee more than 2 times because he has missed a scheduled session or not confirmed a session time, my third email lays out my perspective that there may be a mismatch in what the mentee is able to do (as shown below).

aea365_griffey3

Hot Tip: Don’t rule out a mentoring program because you don’t think you offer the program’s content or focus. I became a CBF mentor in its initial program even though I didn’t necessarily have the business focus I thought they wanted. My match was a mentee who two years later still benefits from my experience in public health and leadership.

aea365_griffey4Mentor isn’t ready: If you have agreed to mentor, respect the commitment or acknowledge that you can’t.

As the mentee, make sure you are getting what you need from the mentor. And if you aren’t getting what you need, don’t be afraid to let the mentor or the mentoring program manager know that. It may be that the mentor really isn’t ready for the mentoring relationship

Hot Tip: it may help you as a mentee to think overall and about each session as answering 3 questions:

  1. What do you need right now?
  2. What do you want to do and why?
  3. How can your mentor help you?

Hot Tip: Being a mentee is as important as your work or schooling. Be proactive in communications, making sure to check your email daily, letting a mentor know what your schedule is, what time zone you are in, and how and when to reach you.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

David Fetterman“Ok, glass.” That’s how you activate Google Glass. I’m David Fetterman and that’s me to the right wearing Google Glass. I’m an empowerment evaluation synergist and consultant, busy father and spouse, and owner of Fetterman & Associates.

Rad Resource – Google Glass: Google Glass is a voice and gesture activated pair of glasses that lets you connect with the world through the internet. You can take a picture, record a video, send a message, listen to music, or make a telephone or video call – all hands free.

Hot Tips – Redefining Communications: Google Glass is not just another expensive (currently about $1500) gadget. It can free us up to do what we do best – think, communicate, facilitate, and, in our case, assess. Here is a brief example.

I said “Ok, Glass,” then “make a call to Kimberly James.” She is a Planning and Evaluation Research Officer I am working with at the W.K. Kellogg Foundation.

Kimberly asked how the evaluation capacity building webinar is coming along. Via Glass, I took a screenshot and mailed it to her so we can discuss it. When a colleague is mentioned, with a few swipes of my finger on the frame, I find a picture on the web, and miraculously remember who we are talking about.

Mid-conversation, Kimberly needed to step away briefly. While on hold, I sent a note to colleagues in Arkansas to ask them to check on the data collection for our tobacco prevention empowerment evaluation.

Kimberly returned to the call and we discussed a recent survey. With a simple request, the display of our results appeared, reminding me what the patterns look like.

Did I mention that I did all of these things while making lunch, picking up my son’s clothes off the floor, letting the dogs out, and emptying the dishwasher?

Later in the day, with a tap on the frame, I confirmed our scope of work with Linh Nguyen, the Vice President of Learning and Impact at the Foundation, while dropping my son off for piano lessons.

Later in the week I plan to use Google Hangout to videoconference with another colleague using Glass. When she connects during a project site visit, she will be able to take pictures and stream video of her walk around the facilities, bringing me closer to the “hum and buzz” of site activities.

Lessons Learned:

Respect people’s privacy – do not wear Google Glass where it is not wanted, will put people off, or will disrupt activities. Do not take pictures without permission. Remove it when you enter a bathroom.

Rad Resources

Hot Tip: Stay tuned for Part II tomorrow when I will cover using Google Glass as an evaluation tool.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Enhanced by Zemanta

My name is Kylie Hutchinson.  I am an independent evaluation consultant with Community Solutions Planning & Evaluation.  In addition to evaluation consulting and capacity building, I tweet at @EvaluationMaven and co-host the monthly evaluation podcast, Adventures in Evaluation along with my colleague @JamesWCoyle.

When I started out in evaluation 26 years ago, I was focused on being a good methodologist and statistician.  After deciding to work primarily with NGOs I learned the importance of being a good program planner.  Employing a participatory approach required me to become a competent facilitator and consensus-builder.  These days, the increased emphasis on utilization and data visualization is forcing me to upgrade my skills in communications and graphic design.  New developments in mobile data collection are making me improve my technical skills.  A recent foray into development evaluation has taught me the important role that a knowledge manager plays in evaluation. Finally, we are starting to understand evaluation capacity development as a process rather than a product, so now I need expertise in organizational development, change management, and the behavioral sciences.  Whoa.

HutchDon’t get me wrong, I’m not complaining.  Every day I wake up and think how lucky I am to have picked such a diverse career as evaluation. But with all these responsibilities on my plate, my toolbox is starting to get full and sometimes keep me awake a night.  How can I manage to be effective at all of these things?  Should I worry about being a Jack of all trades, Master of none?

Hot Tip:  You don’t have to do it all.  Determine your strengths and outsource your weaknesses. Pick several areas of specialization and ask for assistance with the others.  This help may come in the form of other colleagues or departments.  For example, if you think you need help with change management, sub-contract an organizational development consultant to your team.  If you work in an organization with a communications or graphic design department, don’t forget to call on their expertise when you need it.

Hot Tip:  Take baby steps.  If you want to practice more innovative reporting, don’t assume you have to become an expert in communication strategies overnight. Select one or two new skills you want to develop annually and pick away at those.

Hot Tip:  If you can, strategically select those evaluations that will expose you to a new desired area, e.g. mobile data collection or use of a new software.

Rad Resource:  Even if you’re not Canadian, the Canadian Evaluation Society’s Competencies for Canadian Evaluation Practice provide a great basis from which to reflect on your skills.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

We are Bernadette Wright and Ladel Lewis. As evaluators, we often get into the habit of noticing the merit and worth of others’ activities everywhere we go. We may question the effectiveness of marketing materials we see, customer service processes we experience at local stores, surveys we’re asked to complete, and so on. Yet sometimes we neglect to self-reflect on the impact of our own activities—Are we “delivering the goods,” or just delivering? “The shoemaker goes barefoot,” is a saying some have used to describe evaluation researchers who are not evaluating their own work.

Recently, we had opportunity to evaluate our activities as part of our work with the Washington Evaluators (WE) Membership Committee. Other evaluators can use similar techniques to apply evaluation to benefit the activities of their AEA affiliate or their own evaluation work.

Hot Tip: Try creating a logic model diagramming what you are accomplishing and how. During a recent Washington Evaluators board meeting, as the group discussed why and how we want to build membership, Ann Emery sketched a logic model (posted on Twitter) showing inputs, services/activities, outputs, and outcomes. Committees and board members can refine and update the logic model over time. A logic model can help committee staff see how their work fits into the bigger picture, and it can communicate to members what the organization is doing for them.

Cool Trick: Use research evidence and outcomes data to strategically focus energies and resources where they can do you the most good. For example, a 2010 WE Needs Assessment membership survey found that members were highly interested in networking opportunities and mentoring. Membership statistics collected through the WE online membership form provided insight into numbers of active members and lapsed members, numbers of student and professional members, and industries where members worked (e.g., private corporation, self-employed, non-profit, government, university). Comparison with AEA membership data revealed that many AEA members in the Washington, DC area were not members of Washington Evaluators. Based on these results, the Board and Membership Committee developed strategies that aligned with membership interests and identified needs, such as targeted networking events for specific groups like job-seekers and independent consultants.

Rad Resource: While you are visiting Washington, DC for the AEA Conference, consider making an escape from the hustle and bustle to enjoy the wonderful parks that the City has to offer. If you’re looking for fun hiking trails, beautiful woods and views of the Potomac River, and excellent bird and wildlife watching opportunities, Theodore Roosevelt Island and Rock Creek Park always deliver! We are Bernadette Wright and Ladel Lewis, and we approve this message.

How have you used evaluation to ensure you are delivering the goods?

This is the last of three weeks this year sponsored by our Local Arrangements Working Group (LAWG) for Evaluation 2013, the American Evaluation Association Annual Conference coming up next month in Washington, DC. They’re sharing not only evaluation expertise from in and around our nation’s capital, but also tips for enjoying your time in DC. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

No tags

This post comes to you from Valerie Caracelli, Senior Social Science Analyst at the U.S. Government Accountability Office (GAO). I’m serving with David Bernstein as co-chair of the Local Arrangements Working Group (LAWG) for the 2013 American Evaluation Association Conference in Washington, D.C. October 14-19.

Lessons Learned—Do federal managers use performance information? The conference theme, The State of Evaluation Practice in the Early 21st Century, lends itself to the question of whether and how federal managers use performance information to manage for results. GAO’s most recent periodic survey of Federal Managers was recently released under Managing for Results: 2013 Federal Managers Survey on Organizational Performance and Management Issues, GAO-13-519SP.

Lessons Learned—Do evaluations get used? The conference theme also raises the question of whether and how evaluations can make a difference! Stephanie Shipman spearheaded a companion GAO report that focuses on questions from the Federal Managers Survey that address how evaluations contribute to improving program management and policy making. Through case examples, GAO explored barriers that impede use, and strategies that agencies use to get evaluation results used. See Program Evaluation: Strategies to Facilitate Agencies’ Use of Evaluation in Program Management and Policy Making [GAO-13-570]. One finding is that evaluators rely on a body of evidence, rather than a single study, which allows for responding to a broad array of evaluation questions of interest to decision makers. You can download both reports from www.gao.gov.

[AE1] Hot tips for visiting D.C. in 2013. The perspective you gain from a body of evidence led me to think about your visit to our nation’s capital and where you might find the best perspective for viewing the city and its surroundings:

Mall View From Capitol Dome

The view of the National Mall from the Capitol Dome. Image credit: http://dclikealocal.com/dclikealocal/2009/3/23/best-view-of-dc-the-capitol-dome.html

  • Naturally, the Washington Monument comes to mind but that will be closed owing to repairs from the 2011 earthquake!
  • At just over half the height of the monument, you can see a 360 degree panorama of the same views from the Clock Tower at the Old Post Office Pavilion. Donald Trump recently purchased the building but the National Park Service still controls the Clock Tower which will remain open during renovations.
  • Evenings you can enjoy the view from the W Hotel’s 11th floor and P.O.V. Roof Terrace provide panoramic views of the monuments and D.C. landmarks which are illuminated in the evening.
  • Staying awhile? The Washington National Cathedral’s Pilgrim Observation Gallery offers a 360-degree view of the city of Washington and its environs.

For more resources on these and other activities, we encourage each of you to visit the Washington Evaluators Local Affiliate website at www.washeval.org. Washington Evaluators are looking forward to the conference and your arrival!

We’re thinking forward to October and the Evaluation 2013 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Registration is now open! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

No tags

I am Andy Blum, Vice President for Program Management and Evaluation at the U.S. Institute of Peace (USIP), an independent organization that helps communities around the world, prevent, manage or recover from violent conflict.  I recently spoke at a brown bag for the Washington Evaluators about the process of improving my organization’s learning and evaluation processes in general, and creating an organizational evaluation policy in particular. In this post, I’ll provide a few of the takeaways that could be applicable in other contexts.

USIP2Here are three, hopefully generalizable, lessons from the process of crafting the evaluation policy at USIP.

Lesson Learned: Conducting a baseline assessment was extremely helpful. By asking staff their greatest hopes and fears of evaluation, themes emerged, and the findings proved useful as an assessment of where we stood regarding learning and evaluation, and for developing an action plan to improve evaluation.

Lesson Learned: When talking about changing how evaluations are done and used in organizations, you need to manage messaging and communications almost fanatically. The phrase “demystify evaluation” had real resonance. I found myself becoming almost folksy when discussing evaluation. Instead of saying theory of change, I asked, “Why do you think this is going to work?” Instead of saying indicator or metric, I would ask, “What are you watching to see if the program is going well?” Especially at the beginning of an effort to improve evaluation, you do not want to alienate staff through the use of technical language.

Lesson Learned: There is a tension between supporting your evaluation champions and creating organizational “standards.” Your evaluation champions have likely have created effective boutique solutions to their evaluation challenges. These can be undermined as you try to standardize processes throughout the organization. To the extent possible, standardization should build on existing solutions.

Rad Resource: The best change management book I’ve seen: Switch: How to Change Thing When Change is Hard, by Chip Heath and Dan Heath.

Hot Tips—Insider’s advice for Evaluation 2013 in DC: The Passenger is DC’s most famous cocktail spot, but if the weather is good you can’t beat Room 11 in Columbia Heights as a place to sit outside and drink real cocktails.

We’re thinking forward to October and the Evaluation 2013 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Registration is now open! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

No tags

I’m Susan Kistler, the American Evaluation Association’s Executive Director Emeritus and aea365’s regular Saturday contributor. After over three years, this is my last day as your ongoing Saturday author. I’ll be moving to once a month for at least the near future. The other Saturdays will be filled by Sheila Robinson, our intrepid aea365 curator, Stephanie Evergreen, in her role as AEA’s Potent Presentations Initiative (p2i) coordinator, and others from AEA’s new staff team. I am excited to see so many people invested in aea365.

Data Visualization and Reporting (DVR) is near and dear to my heart. The very first post that I wrote for aea365 was on Data Visualization back on January 2, 2010. I am honored that I get to complete up my weekly run by wrapping up the Data Visualization and Reporting week.

Rad Resource – Sarah Rand’s Previous Posts: Earlier this week, Ann Emery included Sarah Rand in her DataViz Hall of Fame. Sarah is part of a team that is breaking new ground, pushing into online reporting and infographic use for research and evaluation. Sarah has written two previous posts for aea365:

Rad Resource – Interview with Sarah Rand: As I explored the online report from Sarah’s team, I wanted to learn more about what it took to produce such a report as well as the drivers behind, and the benefits or drawbacks of, their innovations in reporting. In March of 2013, I interviewed Sarah. The Take 5 Tech video below provides a very brief overview of the report and then a five minute summary of lessons from Sarah.

Lessons Learned: For me, three things stood out from Sarah’s interview:

  1. The composition of the evaluation team may need to adapt to incorporate designers and programmers in order to move into new reporting formats.
  2. There is a need for more knowledge about, and perhaps more tools themselves, for assessing the impact of new reporting formats.
  3. For this group, innovations in reporting have drawn attention to their findings and also to their firm as well, including securing additional grant funding based.

Get Involved: I would like to expand this project into a short series focused on innovators and innovation in data visualization and reporting, in particular for evaluation. This week has given me some great ideas, but if you know of evaluators who are doing leading edge work in this area that you would recommend. please share via the comments on the aea365 blog.

aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. AEA is celebrating Data Visualization and Reporting Week. The contributions all this week to aea365 come from members of AEA’s Data Visualization and Reporting Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

No tags

I am Sean Alan Levin and today I am sharing information about Asana because I have found it incredibly useful in my work.

Rasanaad Resource – Asana: Asana is a project management tool that is free for teams of up to 30 people. I am currently using it both with a project team of 8 and with my partner to keep track of our daily obligations. It touts itself as a “shared task list for teams” but that is underselling its capabilities.

With the work team, Asana manages the work flow. It helped us to set milestones and identify the component tasks and who is responsible for each.

With my partner, we use it as a flexible to do list. I also have to admit that I get a bit of a kick out of checking things off and this feeds my fetish.

Hot Tip – Sort and Filter: Where Asana really shines is in sorting and filtering. You can sort and view tasks by priority, project order, or due date. You can filter by any tags that you have used within or across projects.

Hot Tip – Communicate: Asana is fully integrated with email, so you can receive notices in your email about the activities on Asana and you can post new things to Asana via email. You can also discuss particular projects with your team members. It took me a while to remember the email posting process, but now it is second nature.

Hot Tip – Use the Shortcuts: At the bottom of the page, when signed in, you’ll see “more…” click on the more for a list of the keyboard shortcuts. They are super-useful.

Hot Tip – It’s Free: Well, it is free for up to 30 people on a team, but for the small projects I work on it is totally free. There are also paid versions for larger teams, but I strongly recommend this for consultants and small firms. I have been impressed with the sophistication of the functionality, and the usefulness, for a free product.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I am Jake Lee and it’s time to get organized! I know that many of us say that each new year, and then clean out a closet or read Getting Things Done and check off the resolution for another year. My personal challenge is not around organizing THINGS. My recycling is sorted, my spices alphabetized, my tax receipts categorized. Where I fall short is in the organization of memories and milestones and project events.

I used to keep a leather book filled not only with appointments but also notes in the margins, reminders, and the occasionally clipping from my local (now defunct) newspaper. Once digital options became available, I quickly turned to them and haven’t looked back. But, finding the right product for me has been a frustrating process.

Lessons Learned:

I tried Twitter – adding a note each time I did something that I might want to know at the end of the year. But the 140 character limit was too short, the archive too spotty, and I couldn’t go back and edit or add.

I tried a Blog – David Fetterman wrote on aea365 back in April that keeping a project blog was “particularly helpful at the end of the year when you are trying to construct the annual report.” This, in effect, was what I was hoping to be able to do – draw on the records to construct a report or remember an event or even just compile the Christmas letter to my dear Aunt Phylly. But, it was difficult to see the big picture with a blog, to get a sense for the connections between items over time, and I felt obligated to write more than I really wanted with each entry.

I tried Google Calendar and a number of other calendaring tools – These were the closest to my old appointment book, and had the added benefit of sending me reminders. I still maintain a calendar, but I find that it gets cluttered. It has too much to retain for historical purposes. I want to chronicle personal milestones and professional work products, not dentist appointments and dry cleaning pickups.

Hot Tip – Try Dipity or a Timeline Maker of Your Choice: For 2013, I am trying out using a timeline tool as a chronicle for the year. I have chosen Dipity because it is free, entries are searchable, I can add pictures and links, the visual organization appeals to me, and I can return and update or delete items.

Rad Resource Compare Timeline Tools: The Institute for Public History in Belgium has a very useful timeline tool comparison chart supplemented by a blog post providing narrative depth to the comparisons.

Get Involved: Have you tried to chronicle your projects or personal milestones? What has worked and what has not?

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

No tags

Hello! My name is Aubrey Perry, the Coordinator for Data Collection and Analysis at the Gateway to College National Network. We work with two-year colleges, school districts, and state departments of education to provide opportunities for high school dropouts and underprepared college students to achieve college credentials.

Hot Tip: As an internal evaluator for a nationwide network, I am partially responsible for communicating our data collection practices and processes to individuals of varying backgrounds and skills. I have found that mind mapping is a great way to graphically show the relationships between various principles and tasks.

Rad Resource: For more information on how you can integrate mind mapping with your evaluation projects, check out Dana Dehart’s post on AEA365.

Hot Tip: When looking for mind mapping software, there are a few important questions you should ask:

  1. Is the program flexible enough to accommodate your creative thinking patterns? When most people think of mind maps, the “tree” structure comes to mind where everything branches off a main concept. A good program should do this, and more, by providing the flexibility to design the map that best fits what you’re trying to convey.
  2. Will the program connect to the other programs I use in my office or routines? Mind maps are great to include in evaluation proposals or training materials as a way to graphically represent the processes someone might do. To accomplish this, however, you need a way to link the two programs. Make sure the program you choose allows you to save or export the map as a JPG, PNG, BMP, or PDF file.
  3. Is the program financially feasible for my practice or organization? There are many great programs out there that can help you start mapping. However, these range from free to very expensive. I recommend starting with one or two free programs to see if they meet all of your needs.

Rad Resource: The program that I use for Mind Mapping is xMind. The java-based software has an extensive set of features available for free users, including all of the features I mentioned above. One of my favorite features is the ability to easily embed images to the tree itself, instantly making the mind map more graphic. You can also share the image to the web, making it easy to embed on your virtual training materials. Here’s a mind map showing some of the processes xMind is great for:

Rad Resource: Sometimes the best inspiration comes from others! Join the xMind user group at biggerplate for a network of xMind users that have posted plenty of examples. Also browse the IconArchive for images that can easily embed into your map.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Older posts >>

Archives

To top