AEA365 | A Tip-a-Day by and for Evaluators

TAG | time management

My name is Derrick Gervin and I currently work as a Lead Evaluator at The Evaluation Group (TEG) in Atlanta, Georgia. I work with school systems and nonprofit organizations to improve student achievement. After completing my first six months as a full-time evaluator, I would like to share some tips with other newcomers to evaluation.

Hot Tips:

  • Look inside: Identify your strengths and how they may be used in the evaluation. I realized early on that the field of evaluation was too diverse for me to know everything so I chose to pull from my strengths as a social work practitioner.
  • Relationship building: The more you know about your client and their work, the better you can meet their evaluation needs. I’ve taken to doing a Google search of both the client’s organization and key people in the organization – going beyond just their website helped me to uncover valuable information to assist in my work. Also, I take advantage of opportunities to interact with clients during special events (i.e., career fairs, book festivals, and trainings).
  • Build trust and be accessible: Make commitments and keep them. Ask clients for their input. Set aside time to be available to clients and return calls and emails as soon as possible. I have monthly evaluation meetings to discuss successes and challenges. Also, I spend as much time as possible on site meeting with project staff and observing processes.
  • Get Organized: Find an organizing system that works for you. Also, plan to do as much project management as direct evaluation work. Especially, when projects are at the beginning stage. I’ve found a need to take continuing education classes in time management and the use of Microsoft Excel. I’m constantly searching for ways to maximize my time and work more efficiently.
  • Conceptualization: Explore techniques to assist in conceptualizing planned work and expected outcomes. I regularly visit AEA365 for helpful data visualization tips and conceptualization resources. I really like DoView for creating logic models.
  • Professional Development: Take advantage of opportunities to increase evaluation knowledge and skills. Know your limitations and consult with mentors and other evaluators in the field. I’ve found my co-workers to be a great source for answering and/or talking through challenging evaluation related issues. I participate in monthly lunch and learn sessions, as well as, group conference calls where we discuss and receive feedback on our evaluation projects.

The American Evaluation Association is celebrating Best of aea365 week. The contributions all this week are reposts of great aea365 blogs from our earlier years. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

Greetings! I’m Gabrielle Watson, and I work at Oxfam America, a global humanitarian and development organization working to end social injustice around the world. As the Manager for Campaign Evaluation, I work with policy advocacy teams to assess their campaigns and facilitate learning. We periodically commission external evaluations. They are a big time and resource investment, so we want to maximize their value. We have thought a lot about increasing utilization of evaluation findings, and have a few ideas to share, based on a recent evaluation of our Access to Medicines campaign. My thanks to Jim Coe and Jeremy Smith for their suggestions!

Hot Tip: Design the evaluation to feed into internal deliberation and planning processes. The evaluation should provide adequate data and analysis to support structured reflection & decision-making among key stakeholders. Rather than asking for final recommendations, focus on clarifying the key questions the evaluation should answer. Don’t specify approaches or methods. Instead, invite evaluators to propose relevant approaches. And finally, synchronize your evaluation timeline to existing planning processes.

Hot Tip: Set up an evaluation steering group. This group helps identify critical areas of focus, deliberates on preliminary findings, and actively communicates the evaluation to key audiences. The group should be representative of key stakeholders. I included campaigners, senior managers, and an evaluation colleague who could bring a fresh eye to the methodology and process.

Hot Tip: Maximize interactivity during the evaluation process. The evaluation team shared early findings in an iterative, staged process. By the final report, we had already seen and discussed – and fed into and responded to – all the main findings. There were no surprises. A staged process allowed the evaluation team to adjust the methodology and focus of the evaluation along the way. Early feedback also helped identify gaps and misperceptions, and gave the team a sound understanding of the institutional context for the evaluation. They were better able to orientate later outputs in ways that enhanced their relevance and usefulness.

Hot tip: Plan on having various versions of the final product. From a bite-sized one-page headline findings, to a five-page executive summary, to a 15 – 20 page digest, to the full evaluation report with annexes. A slide deck version lets Steering Group members adapt and present it to different audiences.

Lesson Learned: The commissioning manager must play an active role facilitating information flows, and shaping deliberations, validation processes and dissemination. Budget 5 – 15% of your time during and after the evaluation to disseminate the report and make presentations to different groups.

Lesson Learned: Budget adequate time – probably at least three months from start to finish – for the iterative approach and adequate consultations.

We’re celebrating Advocacy and Policy Change week with our colleagues in the APC Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. aea365 is sponsored

· · · ·

My name is John Paul Manning and I’m an independent data analyst, which means I spend a lot of time working on my computer and it also means that I’m always struggling with fair billing. I’m contributing a favorite tool to shortcuts week because for me it has been a faster, easier, way to do project time tracking (and to stay on task).

Rad Resource – RescueTime (http://www.rescuetime.com/): RescueTime is a SAAS (software as a service) program that runs in the background on your computer and tracks what application or what internet site you are actively engaged with at any given time. It then creates reports that you can review (and if you have employees that you could review with your employees) regarding how one’s computer time is used. It comes in a solo lite version that is free and has met my needs, and a solo pro version for $6/month. Pro allows you to block out particular sites (no Facebook for instance) for a defined period of time to help you stay on task, and it keeps your history longer (lite stops at 2 months).

Lesson Learned – Tracking Helps with Project Planning: I always found it difficult to estimate how much time it really took to draft a report, or to complete an analysis, or to do background research. RescueTime gives me a much better idea of how I spend my time, even when I have to jump in and out of a project due to phone calls and meetings. Its time tracking isn’t perfect, but I’ve been surprised by how accurate it is. I’ve tried time tracking programs where I had to start and stop a timer when I was working on a project, and they just didn’t work well for me when multi-tasking (and I’d forget…).

Lesson Learned – Tracking Keeps you Focused: Just knowing that this is running in the background, and that I’m going to look at what I did at the end of the day, keeps me more on task. It not only is a subtle reminder for me not to go and check weather.com, but also not to flit so much from project to project. Sad but true.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

My name is John Colon and I work for Better Questions Better Answers. I’m amazed by what is available, for free, for my computer. Here are some of my favorite, can’t live without, better than sliced bread tools.

Rad Resources: I’d like to share 9 tools that make my life easier.

10: Gantzilla: Free version of this online service (still in Beta but in good shape) creates great looking and sharable Gantt and PERT charts.

9. Jing: Make short, sharable, voice-annotated screen screencasts (videos of what you are doing on screen) in a flash with this free little cousin to Camtasia. Fantastic for providing instructions.

8. Twitter: I’m careful about what/who I follow (rarely people, mostly news sources) and use it as a way to create my own live news feed focusing on just my interests. AEA’s @aeaweb is a good start for evaluators.

7. Stickies: This free pc utility works like electronic post-it notes on overdrive. Great for reminders. Can stick to sites, desktop, etc. Can set to pop up as reminders.

6. Remember The Milk: I make lots of lists. Even the free version of this lets me make lists, track lists, share lists, checkoff lists. I’ve upgraded to support the developers ($25) and now can use it on my phone too.

5. AutoHotKey: Automates any set of repetitive keystrokes on your computer and can be set to automatically expand abbreviations when typing. I type “tfi” and it writes “Thank you for the great idea.”

4. Bubbl.us: Makes mindmaps online for free. Easy to use, printable, can be exported as jpg.

3. ManyEyes: Very useful, free, suite of data visualization tools.  Comes with great credentials – used by the New York Times and developed by IBM.

2. DropBox: Download dropbox (free). Create a folder in dropbox. Share that folder with a colleague. Now, drop any files at any time into that folder and your colleague can access them from anywhere.

1. ??? I left the number 1 slot open, hoping that you would share your favorite tools via the comments. So that I can learn from you and make life even better.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · · ·

AEA365 began on January 1, 2010. Before we promoted this resource, we reached out to dedicated authors who believed in the project in order to populate the site with starter content. Those who contributed in week 1 wrote for an audience of fewer than 10. One year later we have over 1500 subscribers and are re-posting the contributions from those trailblazers in order to ensure that they receive the readership they deserve. John was kind enough to update his for 2011!

My name is John LaVelle, and I am an advanced graduate student at Claremont Graduate University. An interest (and need) of mine is time management, since it’s so easy to let the day slip away. I’ll be sharing two resources on more effective time management.

Rad Resource: When you’re getting started in evaluation, it’s easy to forget how much time you spend on your project, and that information can be very valuable later on for when you’re estimating costs for future projects. So, I use a free program for the Mac called MyMacTime to help me keep track of my time investments. MyMacTime is available at: http://www.apple.com/downloads/macosx/productivity_tools/mymactime.html

Rad Resource: It’s sometimes easy to get wrapped up in a project and forget about the time. I use a free dashboard app for Apple called Prod Me, which plays chimes or other sounds at regular intervals to help me be more mindful of the time. ProdMe is available at: http://www.apple.com/downloads/dashboard/status/prodme.html

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

My name is Susan Kistler and I am the Executive Director of the American Evaluation Association. I contribute each Saturday’s post to aea365 – the Tip-a-Day Alerts by and for evaluators. While aea365 of course offers my very favorite tip each day, I confess that my attention is sometimes drawn to these other Daily Tips available via email subscription, each of which regularly has content of interest to evaluators.

Rad Resource – Institute of Management Consultants Tip of the Day: AEA member Tom Kelley turned me on to the IMC daily tips. Each one presents a very short scenario or question and provides a concrete tip in response. The questions span the consulting gamut from presentation tips to getting new business. Unlike aea365, where we showcase hundreds of tip authors, the IMC tips are written by a single author/editor, Mark Haas.

Rad Resource – Management Tip of the Day: These come from Harvard Business Review, and tend to be very well-written and blissfully brief, but with the option to click through for an extended article. They usually also include the option to click through to purchase related paid content, but even the stand-alone free portion is very strong. Recently, they’ve covered everything from organizational decision-making to mastering communications.

Bonus! The Harvard Business Review also sends out (via a separate subscription) The Daily Stat “Facts and figures to stimulate thought – and action” which the data diva in me just loves.

Rad Resource – DailyWritingTips: When I was in college, my English teacher told me I wrote as if all composition was some portion of an instruction manual. To her credit and my relief, she conceded that it was a well-written instruction manual, but a manual nevertheless. She was not being complimentary. I have been striving to improve ever since.

This contribution is from the aea365 Tip-a-Day Alerts, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

· · ·

My name is Mika Yoder Yamashita. I am the qualitative evaluation lead for the Center for Educational Policy and Practice at Academy for Educational Development. Our Center has been conducting process and outcome evaluations of the federally funded program, Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP).  This program aims at increasing college access among disadvantaged students.  As we are evaluating programs implemented in several sites, we are beginning to explore the possibility of conducting a multi-site evaluation. Today I will share my Center’s thoughts on how we can effectively approach conducting a multi-site evaluation that uses qualitative data to understand the process of program implementation. Then I will share how we use the literature to guide our data collection and analysis.

Our evaluation utilizes a similar approach to cluster evaluation (W.K. Kellogg Foundation, 1998). We draw upon Davidson’s (2000) approach to build hypotheses and theories of which strategies seem to work in different contexts.  The end goal of our cluster evaluation is to provide the client with a refined understanding of how programs are implemented at the different sites.

Cluster evaluation presents us with the following challenge: How to effectively collect and analyze qualitative data in a limited time to generate information on program implementation. To help us to guide qualitative data collection and analysis, we draw on a literature review.

Hot Tip: Start with literature review to create statements of what is known about how a program works and why it works. Bound a literature review according to the availability of time and evaluation questions. Document keywords, search engines, and decision regarding which articles are reviewed in order to create a search path for others. Create literature review protocols that consist of specific questions.  The reviewers write answers as they review each article. The evaluation team members review two to three summaries together to refine literature review questions and the degree of description to be included. We use qualitative data analysis software for easy management and retrieval of literature summaries. With this information, we draw diagrams to help us articulate what the literature reveals about how a program works and in what context. Using diagrams helps to share ideas across evaluation team members who are not involved in literature review.  Finally, create a statement of how and why the program works in what context and compare these statements with the data from the multiple sites.

Resources: Davidson, E.J., (2000). Ascertaining causality in theory-based evaluation. New Directions for Evaluation, 87, 17-26.*

W. K. Kellogg Foundation (1998). W.K. Kellogg Foundation Evaluation Handbook. Battle Creek, Michigan: Author. Retrieved from: http://www.wkkf.org/knowledge-center/resources/2010/W-K-Kellogg-Foundation-Evaluation-Handbook.aspx

*AEA members have free online access to all back content from New Direction for Evaluation. Log on to the AEA website and navigate to the journals to access this or other archived articles.

This aea365 contribution is part of College Access Programs week sponsored by AEA’s College Access Programs Topical Interest Group. Be sure to subscribe to AEA’s Headlines and Resources weekly update in order to tap into great CAP resources! And, if you want to learn more from Mika, check out the CAP Sponsored Sessions on the program for Evaluation 2010, November 10-13 in San Antonio.

·

I’m Susan Kistler, AEA’s Executive Director, and the author of each Saturday’s aea365 Tip. One challenge we have in the AEA office, that is echoed across those working with groups in almost any context,  is scheduling meetings (or scheduling focus groups or scheduling planning sessions, etc). Challenges include: (1) when we did this by hand with a spreadsheet, even the most diligent made mistakes; (2) it was time consuming and involved a lot of hand-entry, and correction and checking on time zones; and (3) potential attendees varied considerably in terms of the times that they would give (some giving only their convenient times, other giving all possible times including those that would require some wiggling around to make work).

We went in search of a software solution and found a number of time scheduling products on the market. After trying out three, we settled on one that has been a godsend.

Rad Resource: When is Good is a web-based tool that allows you to identify possible times for a meeting across multiple dates, and then send a link to your attendees so that they may choose which times are best. Attendees can click through and identify their available times in their own time zones. So far, these functionalities are common across multiple products and When Is Good does them very nicely in their free version with no signup fees and no registration needed. If you register, you can then create a dashboard where you can manage multiple scheduling events at once, all still free. Thus, the free version solved challenges #1 and #2 from above and was incredibly useful. However, what really made it stand out, for our office, was that when you pay for the premium version, at 5 pounds a year (about US$6.50), your attendees can choose those times that are best, ok, or possible but not ideal. This addressed challenge #3 and has greatly increased the range of times that our colleagues indicate availability, increasing the likelihood that we can identify a common time even across a large group.

Rad Resource: XE.com is an easy to use currency converter site. When Is Good is developed by a family-run software company in the UK (we’ve corresponded with the very nice developer to ask a question or two). Their payment options are all in pounds. Wanting to be sure that we were clear about the extent of our investment, XE allowed for a quick check-in and confirmation that we could, indeed, afford $6.50 for a year of When Is Good.

The above opinions are my own and do not necessarily reflect those of AEA.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

My name is Susan Kistler and I contribute each Saturday’s post to the aea365 blog. This past week, I had the pleasure of attending the annual conference of the Eastern Evaluation Research Society (EERS), an AEA Local Affiliate. At the opening session, George Grob lead a discussion that ultimately focused heavily on tactics evaluators are using for managing their multiple professional obligations, from handling email to making the most of meetings.

Hot Tips: The group offered a number of time and information management tips and I want to thank all who shared their great ideas. Among the offerings:

  • Don’t answer your email until 10:30 AM, instead start your day by diving into a project that you prepared the night before.
  • If you commute, take advantage of the time en-route to think purposefully through a project or plan for the coming day.
  • Remote corners of parking garages can be great places to get work done if you make your car into a mobile office, equipped with the tools and materials you need to succeed.
  • Make appointments with yourself, block out the time in your calendar, and use that time to focus on project work.
  • For those who work from home, create a space in your home where you are conditioned to think certain ways. For instance, set up your office so that when you enter it, you immediately orient towards work because it is filled with the tools of your trade.
  • Wait until someone you respect adopts and recommends new innovations rather than pursuing each new opportunity.

Hot Tip: Get involved in an AEA local affiliate in your area. The wealth of knowledge shared at the EERS sessions and the collegiality made it an event to remember. I learned new skills, built my professional network, and came away with ideas for collaboration. The list of AEA Local Affiliates may be found here http://www.eval.org/aboutus/organization/affiliates.asp and the AEA Online Events Directory lists affiliate events (as well as many others focusing on evaluation and evaluation methodologies).

· · ·

My name is Derrick Gervin and I currently work as a Lead Evaluator at The Evaluation Group (TEG) in Atlanta, Georgia. I work with school systems and nonprofit organizations to improve student achievement. After completing my first six months as a full-time evaluator, I would like to share some tips with other newcomers to evaluation.

Hot Tips:

  • Look inside: Identify your strengths and how they may be used in the evaluation. I realized early on that the field of evaluation was too diverse for me to know everything so I chose to pull from my strengths as a social work practitioner.
  • Relationship building: The more you know about your client and their work, the better you can meet their evaluation needs. I’ve taken to doing a Google search of both the client’s organization and key people in the organization – going beyond just their website helped me to uncover valuable information to assist in my work. Also, I take advantage of opportunities to interact with clients during special events (i.e., career fairs, book festivals, and trainings).
  • Build trust and be accessible: Make commitments and keep them. Ask clients for their input. Set aside time to be available to clients and return calls and emails as soon as possible. I have monthly evaluation meetings to discuss successes and challenges. Also, I spend as much time as possible on site meeting with project staff and observing processes.
  • Get Organized: Find an organizing system that works for you. Also, plan to do as much project management as direct evaluation work. Especially, when projects are at the beginning stage. I’ve found a need to take continuing education classes in time management and the use of Microsoft Excel. I’m constantly searching for ways to maximize my time and work more efficiently.
  • Conceptualization: Explore techniques to assist in conceptualizing planned work and expected outcomes. I regularly visit AEA365 for helpful data visualization tips and conceptualization resources. I really like DoView for creating logic models.
  • Professional Development: Take advantage of opportunities to increase evaluation knowledge and skills. Know your limitations and consult with mentors and other evaluators in the field. I’ve found my co-workers to be a great source for answering and/or talking through challenging evaluation related issues. I participate in monthly lunch and learn sessions, as well as, group conference calls where we discuss and receive feedback on our evaluation projects.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

· · · ·

Older posts >>

Archives

To top