AEA365 | A Tip-a-Day by and for Evaluators

TAG | project management

Hello! My name is Nora E. Douglas and I am a consultant at CNM Connect where we provide evaluation and capacity building services to nonprofit organizations. One of the common challenges I consistently face is managing projects and keeping them moving towards a final deliverable. I have found the following four tips to be useful in the successful completion of my evaluation projects.

Hot Tip #1: Make a Plan and Follow Through

This tip can be attributed to Stephen Covey’s second habit “Begin with the End in Mind” from his 1989 book, The 7 Habits of Highly Effective People. For a different perspective, you can also delve into the world of the Project Management Professional (PMP)®. You need to know where you want to go and make a specific and clear plan for how to do that.

Rad Resources:

Hot Tip #2: Seek out an evaluation champion.

I am always on the lookout for a “champion”; the person within the organization that I’m evaluating that can create enthusiasm and momentum for the project and ensure success.

Rad Resource:

Hot Tip #3: Identify constraints.

The Theory of Constraints was introduced in 1984 for Eliyahu M. Goldratt in his book titled, The Goal. There is always at least one constraint in a project, and identifying that constraint and restructuring the project around it can assist in completing the project successfully.

Rad Resources:

Hot Tip #4: Address tendencies to procrastinate.

I’m guessing we all have procrastinated at one point or another. Procrastination can leave you feeling guilty and anxious about completing the project on time. Two ways I deal with procrastination are to publically commit to getting something done and creating small steps that lead to big progress. Another way to deal with procrastination is to determine the importance and urgency of tasks and which to tackle first.

Rad Resources:

 

Time management matrix as described in Merrill and Covey 1994 book “First Things First,” showing “quadrant two” items that are important but not urgent and so require greater attention for effective time management (Photo credit: Wikipedia / Wikimedia Commons)

The American Evaluation Association is celebrating Labor Day Week in Evaluation: Honoring the WORK of evaluation. The contributions this week are tributes to the behind the scenes and often underappreciated work evaluators do. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

Hello! We are Anne Vo (Associate Director of Evaluation) and Jacob Schreiber (Evaluation Assistant) from the Keck School of Medicine of USC. Anne oversees the Keck Evaluation, Institutional Research, and Assessment (KEIRA) Office and Jacob supports survey data collection efforts at the school.

We survey hundreds of students every week about their curricular experiences as an internal evaluation unit. Like many evaluation teams, we struggle with many moving parts, competing interests in outcomes and processes, and questions about efficiency and cost.

Our team is starting to better understand how individual behavior contributed to or inhibited our ability to collect data from students. These contextual factors are important because they influence the evaluation process, outcome, quality, and the ultimate use of evaluation. We conducted an evaluation process study to begin wrapping our minds around these factors and share what we have learned.

To chart our process—from survey generation to disbursement—we utilized Smartsheet, a project management tool originally designed to assist groups and teams prospectively keep projects on task.

Because Smartsheet offers excellent visualization of workflow, instead of filling spreadsheets with projected dates and deadlines, we recorded the actual time it took to complete tasks. We did this by inputting the time we started and completed a task into the spreadsheet. We also kept track of how long it took other departments to respond to our requests for information, such as lists of instructors needed to create and post surveys.

Lessons Learned:

  1. Smartsheet offers the ability to record and view information about projects and processes in multiple ways. For example, the calendar view indicates various projects that need to be completed, tasks that can be subsumed under each, and the time required to finish them.
  2. Alternatively, Gantt views illustrate when these tasks took place over the course of a week as well as who is involved in their completion. This is a useful tool for demonstrating the flow of a process.
  3. Smartsheet captures data about time by day and week. As such, granular calculations (i.e., by minutes), may not be one of the tool’s strong suits. Still, Smartsheet allows users to export raw data as CSV files so these calculations can be done in other programs (e.g., Excel) or manually.
  4. Charting our survey process in this manner allowed us to pinpoint areas of suboptimal performance. The most compelling finding was that we spent 67% of our time waiting for important information from people in other departments. This helped us alter our own workflows for efficiency.
  5. Smartsheet provides a unique means by which process data can be collected and analyzed; however, we have yet to explore if it can be used to capture information about other factors that contribute to an enterprise’s effectiveness—context, input, and outcome. We hope this post inspires this line of inquiry within the evaluation community.

Rad Resource:

Learn more about Smartsheet here.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi. My name is Jonny Morell. I’m Director of Evaluation at Syntek Technologies, a Principal at 4.669… Evaluation and Planning, and editor of Evaluation and Program Planning.

I recently did an evaluation that made me realize a few things. 1) Project timelines are a type of logic model. After all, they identify program activities and relationships among those activities. 2) Murphy was not quite right. It’s not that anything that can go wrong will, but that some things will go wrong. 3) Things go wrong for different reasons. 4) The reasons things go wrong can provide a lot of useful information about program behavior and outcome. For instance: Did most of the delays come from labor/management conflict, failure to anticipate demands on people’s time, or the demands of a better business climate? Which of these delays were longest, or hardest to resolve? Which affected outcome, budget, or stakeholder expectations? For the important delays, what were the critical incidents that caused the delay? What constellation of small factors combined to cause the delay? I built an entire evaluation around using timelines to answer these kinds of questions.

Lessons Learned: Organizing evaluation around the reasons and consequences of schedule slips provides a lot of useful knowledge that is hard to get by other means.

Rad Resource: I have a blog post that goes into a lot more detail on what I did and how I did it.

Timelines, Critical Incidents and Systems: A Nice Way to Understand Programs

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

 

·

My name is Mike Morris and I’m Professor of Psychology at the University of New Haven, where I direct the Master’s Program in Community Psychology. My research focuses on ethical issues in evaluation, and I am an Associate Editor of the American Journal of Evaluation. The best book I’ve ever read for managing my relationships with stakeholders in an evaluation was not written by an evaluator, nor was it written specifically for evaluators.

Rad Resource: Peter Block (2000). Flawless Consulting: A Guide to Getting Your Expertise Used (2nd ed.). San Francisco: Jossey-Bass. http://www.josseybass.com/WileyCDA/WileyTitle/productCd-0787948039.html

2014 Update:  Flawless Consulting is now in its 3rd edition (2011).

Among organizational consultants this book is legendary. Evaluation is, in my view, one form of consultation, so it’s not surprising that Block’s book is relevant to our work. His discussion of such issues as entry/contracting, dealing with resistance, and managing the feedback of results is invaluable. Central to his analysis is the concept of “authenticity,” which means putting into words what you are experiencing with stakeholders as you work with them. It might sound a bit scary at first, but the more you practice it, the more effective at managing these relationships you become. I also believe that Block’s approach to consulting can enhance the ethical quality of evaluations, especially in terms of helping evaluators identify strategies for raising and pursuing ethical issues with stakeholders.

Flawless Consulting is exceedingly well-written. It probably helps that Block does not have a doctoral degree, since writing a dissertation is a process that can extinguish one’s ability to compose a sentence that anyone would be interested in reading. Flawless Consulting gets very positive reviews from my students. I hope you’ll agree with them. 

The American Evaluation Association is celebrating Best of aea365 week. The contributions all this week are reposts of great aea365 blogs from our earlier years. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hi, my name is Pei-Pei Lei. I’m a survey research analyst in the Center for Health Policy and Research’s Office of Survey Research at the University of Massachusetts Medical School.

As colleagues Michelle Landry and Judy Savageau pointed out in their earlier AEA365 blog on Project Management, Microsoft Office provides some options for no/low cost project management tools. I’d like to share our experience using InfoPath, an MS Office solution for data collection purposes, and how it can also fulfill many project management needs.

InfoPath helps users to collect data electronically from end to end. You can design a form template with the needed data elements, and have the collected data automatically submitted into a designated database. In our survey research work, we also use InfoPath for project management; i.e., we collect employee time sheet data (see below). After employees fill out their time sheets and click “Submit” in InfoPath, the data is automatically submitted to a linked Access database.

Lei 1

InfoPath can be customized to monitor project status, track budget/expenses, manage contracts and related information, assist data collection, and submit an application (see below for a sample template).

Lei 2Lei 3Lei 4

The advantages of InfoPath include:

  • Easy to create – You can create an InfoPath form template from scratch, or convert an existing Excel or Word form into an InfoPath form template.
  • Flexible in form functions – You can embed many useful functions in the template (e.g., data validations, range limits, conditional formats for certain work flow, automatic time stamps).
  • Convenient to distribute form templates and collect data – You can distribute form templates to and receive data from multiple recipients using MS Outlook. With access to a shared drive, users can save form templates to their computers and submit data from there.
  • Allows different forms for the same project to be collected simultaneously – Multiple InfoPath templates can be linked to one database, allowing users to view/complete different forms specific to their roles, and all submit into the same database.
  • It’s free – if you have these versions of MS Office.

Lessons learned: InfoPath can be very useful in project management processes; it is easy to build and distribute for collecting and submitting data into a database (such as SQL, Access, or SharePoint). You will need InfoPath Designer to create the form template. Users will need InfoPath Filler to enter data unless the database is deployed on SharePoint, which allows users to fill and submit data in a webpage from a computer or mobile device.

Rad Resources: Introduction to Microsoft InfoPath 2010 and Project plan template

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hello. I am Valerie Konar from the University of Massachusetts Medical School’s Center for Health Policy and Research and Project Manager for the evaluation of the Massachusetts Patient-Centered Medical Home Initiative. I’d like to share some tips about managing a complex multi-stakeholder project.

When the project began, these three groups operated in separate silos:

Konar 1

When the evaluation team needed data about the project to complete its evaluation, communication with the implementation team became essential as they were the gatekeepers to the study group of primary care practice sites. Members of the implementation team wanted the evaluation results to assist them with quality improvement, so the evaluation team had to determine which results could be shared without compromising the evaluation. Finally, external stakeholders wanted to learn about the project’s progress, so the evaluation team produced evidence-based reports for them.

As the project progressed, data and reports began to link and overlap among the three groups. The evaluation team remained in control of its data at all times and determined what information could be shared and how to share it.

Konar 2

Lessons Learned: A single project manager working with all groups (i.e., teams and stakeholders) can:

  • be the consistent contact and can answer questions regarding requests and reports. This allows teams to focus on their own tasks.
  • maximize data collection response rates by simplifying and communicating the ask, monitoring responses, using the implementation team to engage the study group, and sending reminders.

Hot Tips: Create an overview of the evaluation activities including names of people involved, data needed to answer evaluation questions, data collection tool purpose, description of administrative activities, timeline of requests and report deadlines. This overview can be used by other evaluators to appreciate all components of the evaluation. It can explain data requests to the study group so they understand where their time and effort is going and how they may benefit from the information. External stakeholders can see the breadth of data needed for a comprehensive evaluation.

Engage and communicate with external stakeholders and implementation staff. Invite implementation staff to evaluation meetings to discuss administration and results. Inform stakeholders where they can find both formative and summative evaluation reports.

Incorporate evaluation activities and timelines in all project correspondence, websites and documents:

  • List (and continually update) requests and report distribution
  • Post evaluation activities, timelines and reports on the project website
  • Present results in webinars with external stakeholders

Insert evaluation activities into project contracts to ensure compliance and instill evaluation as integral to the project.

Provide evaluation data to study subjects. Study subjects want to know how they are doing and information engages them in the process.

Rad Resource: Create a Gantt Chart timeline that everyone can read and understand.

The American Evaluation Association is celebrating Massachusetts Patient-Centered Medical Home Initiative (PCMHI) week. The contributions all this week to aea365 come from members who work with the Massachusetts Patient-Centered Medical Home Initiative (PCMHI). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

Hello!  We’re Michelle Landry and Judy Savageau from the University of Massachusetts Medical School’s Center for Health Policy and Research.

As Sean Allen Levin suggested in a recent aea365 post, organizing an evaluation project or multiple projects can be daunting. The details of timelines, responsibilities, and deliverables can easily be lost if they are only in the project manager’s head. As with any project, maintaining a quality and structured work environment benefits the entire team and clients. To expand on Sean’s blog, we’d like to share hot tips and lessons learned during our recent review of project management tools and how it’s brought to light not only the vast sea of options, but also what our needs/wants are from the tool.

Lesson Learned: Work backwards to determine your specific needs. What outputs are most useful? What information is needed to report regularly? How large is the project or how many projects need to be captured in the database?

Lesson Learned: Determine if you have a budget or spending limit to support a new tool. There are affordable options that aren’t readily obvious. However, if you have the budget, there are options to satisfy your every whim.

Hot Tip: For no/low cost options, look to Microsoft Office. This is affordable because most users already have the software; e.g., MS Excel, MS Access, and MS Project, which have onsite relational databases with ranges in user-friendliness/abilities. A number of software vendors sell robust project management tools, but they come with a price tag. We reviewed tools used by our university colleagues; e.g., Quickbase (now piloting) and Journyx. Many others are available with websites offering comparisons among the applications.

Lesson Learned: Review your needs against the software’s options. Many websites allow testing project management tools through a virtual tour. Take advantage of this; it’s best to see how user-friendly the software is before purchasing.

Lesson Learned:  Each application has budgetary implications, so if, of necessity, you’re budget conscious, check into the vender’s software offerings. Is it a one-time cost or license needing annual renewal? Does it require a monthly user fee?

Hot Tip: Customize, customize, customize… Most software packages are customizable. Do not take it “out of the box” and assume that’s all you get. Many vendors offer customization options to meet your needs. Customizing can take a few rounds as test-driving one change often uncovers additional changes. Customizing the software may reduce frustration and better meet your needs. See if there’s someone “in-house” who can customize your software before paying the vendor – a great budget saver!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Howdy!  I’m Tom Ward, and I am a faculty member of the U.S. Army Command and General Staff College at Fort Leavenworth, Kansas.  In that capacity, I teach critical thinking, ethics, contracting, logistics, and “writing to persuade.” My passion, however, is knowledge management, and using KM to improve decision making.  My tip today is about providing project leadership.

We sometimes forget that very competent adults still require leadership when participating in group endeavors. The more they understand the intent of a project and the desired end state of a client, the more they are able to bring their own talents to bear.  This requires not only vision on the part of leaders, but a clear and understandable communication of that vision – preferably in written form.

Hot Tip:

  • Project leaders must understand the environmental context of the project, visualize the steps required to complete the project, and communicate that visualization effectively.  For example, if the deliverable product is a written report or graphic presentation, examples of similar deliverables provide excellent visualization of “this is where we are going.”  “How we will get there” is crucial as well, but may part of the design phase of the problem solving process; still, clearly describing the process of “how we will decide how we get there” enables unified effort and enhances the ability of individuals to contribute effectively.  Leaders who provide clear frameworks for task accomplishment and then provide the required resources for individual and group success gain a reputation not only as reliable producers for clients, but “favorite bosses” of project group members.

The American Evaluation Association is celebrating the Business, Leadership, and Performance TIG (BLP) Week. The contributions all week come from BLP members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

I am Priya Small, an independent public health evaluation consultant, and today I will talk about lessons I have learned from riding the steep learning curve. Many times we may tend to devalue our early learning experiences as evaluators. But as this is a time of rapid growth, sharing these lessons is profitable.

Lesson Learned: Listen more, speak less. Observe more, take less notes. Pay especially close attention to how veteran evaluators manage meetings with stakeholders, build relationships, and resolve potential conflicts. Such learning opportunities may be scarce later.

Lesson Learned: If you are waiting for that job opportunity to come through, consider working on a pro bono project. This can offer a valuable opportunity to craft a useful evaluation in your interest area. Pro bono work can provide immediate motivation to sharpen and fine-tune your evaluation skills and network. The best way to perfect your skills is “on-the-job.” Look for committed managers who are open to seeing the value of evaluation. Negotiate a written agreement that clarifies project responsibilities.

Lesson Learned: Compete less and cooperate more. Team work has great potential to produce optimal outcomes. Don’t hesitate to support other evaluators by commenting on their blogs, joining discussion groups, etc. In a very organic manner, this has provided me with opportunities to collaborate.

Hot Tip: Participate in the AEA discussion group and other evaluation discussion groups on LinkedIn. The atmosphere is supportive and collaborative, and the format is efficient.

American Evaluation Association Linked In account image

Join AEA on Linked In

Rad Resource: The AEA365 blog! It is easy to get “tunnel vision” as we get caught up in our own projects. This can stifle creativity and innovation. Read and reflect on AEA365 blogs posts regularly to see a panoramic view of the vastness of the evaluation landscape. You might just pick up other “Rad Resources” on the way. Endeavor along with me to never stop learning from others. And I encourage you to also reach out by posting your own lessons learned on the AEA365 blog!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

Hello, my name is Nicole MartinRogers. I am the Survey Research Manager at Wilder Research, in Saint Paul, Minnesota. Wilder Research manages projects that range from small evaluations for local nonprofits to multiyear statewide evaluations of complex interventions.

Hot tip: Project management (the discipline) offers great tools that will help keep your projects on track, for example:

  • Work Breakdown Structures, used to plan a project by breaking it down into smaller, more manageable tasks.
  • Three-Point Estimation, a strategy for creating realistic cost estimates by accounting for variability using experience, historical information, and expert judgment to define an optimistic estimate, a pessimistic estimate, and a most likely estimate.
  • Network diagrams and Gantt charts, tools that help illustrate a project timeline and contingencies between tasks, so you can easily show how a delay in one aspect of the project will result in changes to the entire project timeline.
  • Agile or Scrum methods for regular team check-ins to keep your team and project on track.

Lessons learned: Don’t feel you have to adhere strictly to the templates and documentation associated with project management (the discipline). Instead, try out a few tools, take what works for you, and modify it to address your specific needs.

Try working with your team to develop a Work Breakdown Structure or network diagram. It will help them understand how their part fits into the big picture. Also, work with your client to finalize these documents, and get client sign off as a formal step in the project.

Team members, clients, and other project stakeholders are easier to manage and more satisfied when your evaluation project is completed on time and within budget, and when it delivers all of the agreed upon products. This can be achieved by carefully planning and managing resources and expectations throughout the course of the evaluation.

Rad Resource: Google is a great source for more information about any of the specific tools described above. Two books that are also helpful: PM Crash Course, Premier Edition: A Crash Course in Real-World Project Management, by Rita Mulcahy; and Project Management for Dummies, by Nick Graham.

Twin Cities Hot Tip: When you’re in town for the AEA conference this fall, you should check out some of Minnesota’s beautiful state parks. William O’Brien is less than an hour drive from the Twin Cities and has great hiking trails and views of the St. Croix River.

The American Evaluation Association is celebrating Minnesota Evaluation Association (MN EA) Affiliate Week with our colleagues in the MNEA AEA Affiliate. The contributions all this week to aea365 come from our MNEA members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Older posts >>

Archives

To top