Hello Colleagues! My name is Judith Kallick Russell. I am an independent evaluation consultant in civic engagement, community development and peace building. My clients are national organizations (NGOs) and international organizations (UN agencies, international NGOs and foundations). In my work, I have found that it can be challenging at the report writing stage to provide findings and recommendations which are easily translated into actions for clients. The following are some ideas to address this.

Hot Tip: Include boxes or comments on the side in the findings section which suggest questions for reflection. There may be findings which raise questions you feel require further consideration by the client. Including thought provoking questions or comments in the report – visually separate, but linked to a finding – can encourage the client to explore the issue after the consultancy is completed.

Hot Tip: Frame your recommendations in stages or levels. Some organizations are not ready or able to make big changes at the moment of the evaluation. Once you learn from them what they feel capable or interested in doing, you could structure your recommendations providing options with stages or levels. For example, you might want to describe recommendations for a particular issue according to good, better and best.

Hot Tip: Make time for dialogue when finalizing the report. Consider establishing a process for finalizing the report in the very beginning. You might want to gain informal feedback from a few key stakeholders. Then provide a finalized draft to a representative group within the organization. Maybe conduct a workshop about main findings and recommendations, encouraging participation and collective thinking to deepen their understanding of the issues they face. Incorporate all input into the final report as you see fit. Be sure to focus who you ask input from and what input you are asking for, give clear deadlines, and phrase communications in a way where you are not stuck waiting for someone’s response.

If you want to learn more from Judith, check out the sessions sponsored by the Independent Consulting TIG on the program for Evaluation 2010, November 10-13 in San Antonio. Hope to see you there!

communication, recommendations, reports

Greetings All! I’m Gwen Fariss Newman and I’m filling in this Saturday for AEA’s Executive Director while she is on vacation. You may know me as AEA’s Communications Director, a new post as of two years ago. In that time, we’ve revamped AEA’s monthly newsletter and recently introduced a feature that showcases how AEA members are communicating with clients and stakeholders.  Because e-newsletters are one of the easiest and most low-cost ways to get the word out about your evaluation work, I’m devoting today’s post to them.

Rad Resource: Printed newsletters are costly to produce, expensive to mail, hard to track and near impossible to update if sudden changes occur. Electronic newsletters are sent with the touch of a button, going directly to the mailbox of recipients who can instantaneously opt-in or opt-out. This means more targeted mailings, more up-to-date mailing lists and a far more economical mode of production. You can incorporate photos, personalize the contents, make last-minute updates and time delivery. There are ready-made templates or you can create your own. AEA uses ConstantContact. Other popular choices include GraphicMail, iContact and emailbrain.

Hot Tip: Pictures often speak louder than words. Whenever possible, look for images that illustrate the article, identify those quoted, or offer a higher profile to the piece. Quick test: scan the page of a newspaper or magazine. Do you gravitate to the blurbs with images or without? There are almost always ways to visualize the content – and make the piece more appealing, memorable and  likely to be read. You can even run a photo solo with great results.

Hot Tip: Don’t just go green, go evergreen. E-newsletters cut paper costs and production time. But a link to your website can make the content readily visible, easily accessible, and permanently available. Visit AEA’s archive page and see how your e-newsletter can become a ready resource that you can download, forward to others, or refer to at any time.

Lesson Learned: Track reader trends. Most e-newsletters services offer statistics that show who’s reading what. AEA’s newsletter open/read rate is higher than the industry average – thank you! – and readership stats can help you tailor content to your readers’ needs. We run a Top 10 list every February.

Hot Tip: Offer a diversity of articles, views, and perspectives. Remember, communications is a two-way street. You (1) share with readers (2) the news that they want/need to know. This will help you better connect with your audience and ensure a more satisfying experience for both parties. Imitation is the sincerest form of flattery. So, if you have a newsletter that you’d like to share with AEA members or other ideas for AEA’s newsletter, email me at gwen@eval.org.

communications, newsletter, photos

My name is Elizabeth Hutchinson and I am a Monitoring and Evaluation Specialist with Land O’Lakes International Development. My work focuses on evaluating United States Government-funded agriculture and food system development programs. Most people in the U.S. know of Land O’Lakes for its branded butter, but the division I work for has 30 years experience implementing international development programs that generate economic growth, improve health and nutrition, and alleviate poverty through market-driven business solutions.

Evaluation activities are integrated throughout the lifecycles of our programs – it is critical for our funders and because ongoing feedback enables us to quickly make programmatic course corrections. We often work with external host-country consultants who bring valuable localized knowledge and evaluation expertise to support our programs. I’d like to share a few lessons learned to improve the practice of working with local evaluators in international settings.

Lesson Learned – The capacity of local consultants varies greatly: Solicit proposals using RFAs from a variety of sources including universities, private organizations, and individuals. Keep in mind that some evaluation skills and expertise may overlap with other fields (i.e. economics, statistics, sociology, public health) which may be beneficial, depending on the evaluation question(s), subject area, and selected methodologies. Reviewing samples of past work (ie. reports, protocols, data sets) and checking references are both good ways to understand the skill sets and capacity of local consultants.

Lesson Learned – International settings can offer unique evaluation challenges: The political/social context in which the evaluation is being conducted may have implications on the quality and timeline. Use knowledgeable local informants to ensure your methodology, questions and timeline are appropriate to the local context. For example, you may not want to conduct household surveys in the month before a significant national election. Allowing a cushion is also critical, particularly if you have strict funder-mandated deadlines. We have found staggering deliverables (drafts, etc) throughout the project timeline to be helpful. There should be ample time built in to make significant changes before a hard/final deadline.

Lesson Learned – Open and ongoing communication is critical: Ensure that expectations around communications are clear and agreed upon upfront. Be explicit about the language you would like deliverables submitted in (i.e. most U.S. organizations likely expect reports in English). The submission/delivery mode should also be determined since sharing documents internationally can be difficult – identify the most appropriate way to share large electronic files, photos, or hardcopy reports, etc. since it can have resource implications. For example, e-mail submission may be more cost- and time-effective than requiring a printed/bound hard copy shipped from the field.

Related Resources:

• IDEAS, International Development Evaluation Association
• RealWorld Evaluation: Working Under Budget, Time, Data, and Political Constraints (SAGE) by Michael Bamberger, Jim Rugh, and Linda Mabry.*

Would you like to discuss issues of capacity building in international development with Elizabeth? She’ll be presenting at Evaluation 2010, the annual conference of the American Evaluation Association this November. Search the online program to find Elizabeth’s roundtable or information about over 500 other sessions.

*AEA members receive 20% off of all books from SAGE when ordered directly from the publisher. If you are a member, sign on to the AEA website and select “Publications Discount Codes” from the “Members Only” menu to access the codes and process.

consultants, funders, international

My name is LaMarcus Bolton, and I am the American Evaluation Association’s Technology Director, as well as a doctoral student in Industrial-Organizational Psychology. AEA’s Executive Director, Susan Kistler, is on vacation and I’m filling in with this Saturday’s staff post. Today, I wanted to discuss some of the great strides technology has made in terms of helping to improve communication, in particular focusing on quick, easy, and low-cost/no-cost tools that can help evaluators to keep their costs in check and maintain open communication with stakeholders and clients.

Rad Resource: Despite how far we have come in terms of communication mediums, the good ‘ol telephone is still often the medium of choice. Skype, allows one to make free calls to other Skype users, and has very competitive rates for both national and international calls for non-Skype users. Skype even allows one to create conference calls, although with a limited number of participants. However, if you do not want to rely on voice over Internet protocol (VOIP) technologies like Skype (which, by the way, typically are not as clear as traditional landlines), services such as FreeConferenceCall.com in the US may be worthwhile. I have used FreeConferenceCall.com on a few occasions and have always had pleasurable experiences.

Rad Resource: Although the wide availability of scanners and PDFs have made fax machines less popular, there is sometimes still the need to send or receive a fax. Services like eFax give users a free (though non-local) number that can accept faxes and send them directly to your email address. Unfortunately, to send faxes requires one to upgrade to their eFax Plus service. To address this, services like Fax Zero allows one to upload Microsoft Word and PDF files to send to any fax number in the country. Though these options are reliable, and environmentally friendly, I would caution against using them to send or receive sensitive information.

Rad Resource: If, like myself, you try to stay on the cutting-edge of technology, there have been great strides here as well. In addition to phone calls, Skype also allows for free video calls. This allows one to talk face-to-face with friends, family, and colleagues, by adding a small hint of personalization. Though, for quick and efficient communication, technologies like instant messaging (IM) are king. Though, the problem with instant messaging is that everyone seems to have a different preferred client. For example, popular here within the US are AOL Instant Messenger (AIM) and Google Chat (Gchat), but internationally, Windows Messenger is often preferred. To address this, software like Trillian and Digsby for Windows, and Adium for Mac, allow one to combine all IM services into a single and easy-to-use interface.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

communication, skype

Hi, my name is Heidi Gegax and I am a Minneapolis-based independent evaluator.  In my business, I focus on clients who work with youth in out-of-school time programs.  Most of the work I do is solo, which leaves me craving professional development without the cost and time needed for conferences or workshops.

Rad Resource: Create a Consultants Collaborative

I wanted meaningful interactions with other independent evaluators.  As I talked to another consultant, I asked her if she wanted to get together occasionally to talk shop and share professional insights.  She told me of a collaborative she belonged to years ago.  It was a group of women qualitative evaluators, and they called themselves “Quality Women Who Don’t Count”.  Since I wanted to network with evaluators and youth development practitioners as well as men, I decided that name would not be too inclusive, but I did want to start a collaborative.  She and I shared our list of contacts, and invitations were emailed to a small group, inviting all to a centrally located coffee shop for the Twin Cities Consultants Collaborative.  All were interested and we began our first meeting by introducing ourselves and talking about our areas of expertise.

After a few sessions of talking about what we needed professionally from such a group, we created a list of topics and each member selected session(s) to arrange a guest speaker or to lead us through the discussion. Those meetings included social networking, establishing your business as an LLC, legal contracts and work/life balance.  When we looked at the topics we had chosen, the focus fell into the following topics:

• The business of independent consulting
• Professional development
• Tools of the trade

What is really cool about this group is that we have enough variety in our areas of expertise that we complement each other well.  Many of us have submitted proposals together.  Expanding our network has taught us many things about the business of consulting as well as provided an outlet for professional development.  The group started as consultants around youth development, but we are currently expanding this network to include other areas such as marketing and web development.   This will enhance everyone’s experience and round out some professional needs as well as satisfy our need for “critical mass”.

I have come to depend on this network of fabulous individuals who challenge and support me, enlighten my work, and make me feel a little less lonely in the world.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

consultants collaborative, Networking

My name is Linda Meyer and I am an applied developmental psychology graduate student at Claremont Graduate University. I am going to give you a few tips on helping clients develop goals and objectives that can help you plan an evaluation.

Knowing programs’ goals and objectives is useful for creating logic models for evaluation, but new programs or project managers who are unfamiliar with evaluation may not know how to (a) develop goals and objectives or (b) easily explain their goals and objectives. The problem I run into most often is that they may have goals and objectives, but for whatever reason these goals and objectives do not aid the evaluation planning process.

Hot Tip: Create new goals – don’t try to change old ones. Especially if you are an external evaluator, program management may be weary of someone coming in and telling them how to run their programs. Clients often have mission statements or goals that they use for program promotion, but these may not be useful for evaluation. To work with program management and get the evaluation started on solid ground, I start by explaining how to write goals and objectives for the purposes of the evaluation. Often, they end up being similar to the goals or objectives the client originally had for their program, but are worded so that a clear and straightforward logic model and evaluation plan can be created. Clients are much more willing to help you write something that will help plan the evaluation than they are to amend their program’s stated goals and objectives. You get the information you need and they don’t feel like you are taking over their program.

Hot Tip: Use a visual aid to help define goals and objectives. Explaining that goals are abstract and broad and that objectives are concrete and narrow is not enough to help your client develop goals and objectives that will be useful to the evaluation. I usually do a demonstration using simple paper-and-pencil: I draw a large circle and write “goal” in the middle, explaining that goals are like a big, gaping hole that the program is trying to fill (or the need they are trying to meet). There isn’t a shovel big enough that can fill it with one scoop. Then I draw smaller circles in the “goal” circle and write “objective” in each, explaining the relationship between the two terms and how it is easier to fill multiple smaller holes sectioned off from the goal. Objectives are easy to measure and when achieved, contribute toward the broader goal. From there, I explain what outcomes are, which helps later on when you are picking measures. You can use whatever metaphor you like, but people tend to pay more attention when you draw a picture.

Rad Resource: The CDC has their own model for creating objectives, using the acronym “SMART” to help people come up with Specific, Measurable, Achievable, Realistic, and Time-bound objectives: http://www.cdc.gov/phin/communities/resourcekit/tools/evaluate/smart_objectives.html. The website has a link to a worksheet in a Microsoft Word document you can use if you have trouble getting clients to define appropriate objectives.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

goals, objectives

My name is Susan Kistler and I am AEA’s Executive Director. I contribute each Saturday’s post to the aea365 blog. AEA is striving to help evaluators know about evaluation funding opportunities, working to improve on two fronts:

1. Listening and Identifying: Improving our processes for knowing about funding opportunities
2. Disseminating and Communicating: Improving our processes for letting the evaluation community know about funding opportunities

Hot Tip: To learn about funding opportunities for evaluators, subscribe to AEAWEB: Headlines and Resources. AEAWEB includes a range of headlines and announcements of interest to evaluators, and is where we post an announcement of each identified funding opportunity. The easiest ways to receive AEAWEB are via one of the following:

• Weekly Email Compilation: Receive AEAWEB as a single email compiling the week’s announcements, sent each Sunday – sign up at http://www.eval.org/AEA.signup.asp
• EVALTALK: Subscribe to AEA’s listserv, EVALTALK, where the list is posted each Sunday http://bama.ua.edu/archives/evaltalk.html

Hot Tip: If you are on Twitter, follow @aeaweb to receive headlines and resources announcements in real time rather than as a weekly compilation.

Hot Tip: Here are four opportunities posted this past week, to give you an example of the announcements included on AEAWEB.

• Funding: Impact 2.0 – grant opportunity for evaluating web 2.0 in Latin America http://ow.ly/1RA7n
• Funding: Evaluating Oil Spill Disaster Response? Grants Available Through NSF http://ow.ly/1RBow
• Funding: Grants available for graduate students evaluating Head Start Programs for their dissertation research http://ow.ly/1RzkFFinally
• Funding: Establish National Center on Program Management and Fiscal Operations from US DHHS http://ow.ly/1RBFh

If you know of a funding opportunity for evaluators, please share it via an email to headlines@eval.org.

AEA resources, Funding, grants, headlines

I’m Jim Hammerman, and I Co-Direct the Evaluation Group at TERC, a non-profit, primarily grant-funded math, science and technology, education research, development, and evaluation company in Cambridge, MA. I’m also a current member and former Chair of TERC’s Institutional Review Board (IRB).

Several of us recently noted on AEA’s LinkedIn group that there are at least three ways to address the issue of Institutional Review for nonprofit organizations:

1. Partner with a university and use their IRB process;
2. Contract with a commercial IRB;
3. Develop the capacity in-house, establishing your own IRB.

Over the course of three days on the aea365 blog, we’ve been sharing a few lessons learned about pursuing these approaches. I’m going to talk about option 3.

Lessons Learned: As a mid-sized non-profit of about 120 employees, several years ago we found it both necessary and advantageous to establish our own IRB. It took a bit of effort to complete the paperwork and establish the policies to obtain and maintain a Federal Wide Assurance (FWA) number through the NIH Office of Human Resource Protection (OHRP).

We started with documents borrowed from collaborators and with advice from some of our own staff who had served on university IRBs. Initially we had paper-based record-keeping, but setting up electronic systems for archiving and tracking IRB review status has been very helpful.

Having our own IRB allows us to review and oversee projects for which we’re prime, or for which our evaluation subcontract is the only real research (e.g., the prime is doing development). But arrangements can be flexible. Sometimes we obtain review for our portion of a project through a university or other institution with their own IRB when they’re the prime; sometimes our IRB oversees research of our partners if they don’t have their own IRB. For liability reasons, we don’t conduct reviews for folks with no affiliation with TERC.

Having our own IRB allows staff to work with peers who know and understand the nature of the work when developing procedures and informed consent documents. Rapid modification of procedures or consents is also easier with an in-house IRB – e.g., when we encounter unanticipated issues in some of our cutting edge work in online learning environments or with out-of-school library or community action groups. Experience as a member of the IRB can also help junior researchers become familiar with a range of study types and issues, building internal capacity. Having your own IRB can provide a number of benefits for your organization.

Resources: IRB guidelines and requirements regarding membership, meetings, initial and continuing review procedures, record-keeping, and the like can be found in the Code of Federal Regulations 45CFR690.101-124 available through the NSF (http://www.nsf.gov/bfa/dias/policy/human.jsp) or NIH (http://www.hhs.gov/ohrp/).

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Institutional Review, IRB

My name is Stephen Meyer and I am a Senior Research Associate with RMC Research Corporation in Denver, Colorado. RMC Research conducts research and evaluation related to a wide range of education topics.

Colleagues and I recently noted on AEA’s LinkedIn group that there are at least three ways to address the issue of Institutional Review for nonprofit organizations:

1. Partner with a university and use their IRB process;
2. Contract with a commercial IRB;
3. Develop the capacity in-house, establishing your own IRB.

Over the course of three days on the aea365 blog, three of us are going to share a few lessons learned about pursuing one of these approaches.

Most of RMC’s research and evaluation projects require that we seek Institutional Review Board (IRB) approval. We typically work with a university IRB when one is a partner on a particular project, but have found it rare that a university is willing to conduct an external review unless it is in such a partnership role.

Lessons Learned: Recently, for a study involving prison inmates, I approached several IRBs at universities that conduct related research and found that they either were unwilling to conduct reviews for outside organizations or that they lacked the capacity to address the additional requirements related to reviewing studies involving prisoners. While a handful of commercial IRBs also lacked the capacity to review studies involving prisoners, most were able to do so. The costs were substantially higher than what we had been charged by university IRBs–initial review costs ranged from about $1400 to nearly $3000, with additional costs for (annual) continuing reviews, protocol amendments, reviews of Principal Investigator credentials, and reviews of advertisements used for subject recruitment. Although we have had a negative experience with one commercial IRB (related to inconsistent turnaround times and billing practices), our overall experience have been positive. Relative to my experience with university IRBs, I have found that commercial IRBs tend to: have shorter review turnaround times (as little as one week), be more responsive to questions, and have clearer procedures for communication and documentation of project protocols.

Rad Resources: Some options for locating a commercial IRB are below:

• Citizens for Responsible Care and Research (CIRCARE), a human rights organization dedicated to the protection of human subjects in research and medical treatment, maintains a list of commercial IRBs. http://www.circare.org/info/commercialirb.htm
• The Office for Human Research Protections (OHRP) maintains a searchable database of registered IRBs. OHRP-registered IRBs have provided assurances that they will abide by federal regulations related to protection of human subjects. http://www.hhs.gov/ohrp
• The Association for the Accreditation of Human Research Protection Programs (AAHRPP) maintains a list, by state, of the organizations that have met their accreditation standards. http://www.aahrpp.org

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Institutional Review, IRB

My name is Benjamin Cohen and I am Director of Evaluation at the Center for Schools and Communities, a division of the Central Susquehanna Intermediate Unit. We currently coordinate several interventions and initiatives for school-age youth that are funded by federal, state and private funders. My current focus is the evaluation of a state-wide bullying prevention program operating in over 100 schools and 20+ counties.

My colleagues and I recently noted on AEA’s LinkedIn group that there are at least three ways to address the issue of Institutional Review for nonprofit organizations:

1. Partner with a university and use their IRB process;
2. Contract with a commercial IRB;
3. Develop the capacity in-house, establishing your own IRB.

Over the course of three days on the aea365 blog, three of us are going to share a few lessons learned about pursuing one of these approaches. I’ll be talking about partnering with a university.

Lessons Learned: In some cases we need to utilize an IRB to document the ethical conduct of our work and to be compliant with related regulations. Since our organization is a smaller human services provider and not a research organization, we do not have the capacity to build our own IRB, and thus seek university partners to fulfill that role.

University IRBs have offered us the convenience of a packaged review process for our research and evaluation projects. And because we use university IRBs in concert with a faculty partner, we can often rely on those faculty to navigate our project proposals through the IRB’s waypoints, without having to directly respond to IRB members. Having a faculty member on our evaluation teams permits us to focus on issues directly related to project operations, data collection, logistics and other administrative issues. Furthermore, the training that university IRB members typically have lets us have confidence in their expertise in informed consent issues and procedures, and gives us an assurance that we are conducting our work according to high standards of peer review.

Nonetheless working with a university IRB has some disadvantages. A practical one is covering universities’ overhead costs on a project budget. Universities, in short, can be expensive to include as project partners. Among universities in our region, none appear to offer their IRBs on a fee-for-service basis, therefore university faculty must be included as contracted project staff. Furthermore, university schedules for IRB review do not always correspond to project deadlines, nor do project stakeholders always understand the nature of university IRB protocols or how informed consent issues can manifest in the conduct of a project. For these reasons we are also looking into private IRB services that exist outside universities.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Institutional Review, IRB, University