My name is Shobha Mittal and I am an independent consultant. Seeking a higher response rate on the surveys has always been tricky in the field of evaluation.

Lessons Learned: Although there are many good references available to help evaluators impact the response rates, I have found that thinking like a respondent before seeking their response improves the likelihood of success. From a respondent’s perspective there are two major concerns (Besides of course having the inclination or time to respond). First is the technological comfort of responding to a questionnaire electronically versus the hardcopy response, and the second is the fear of consequence which may follow an honest response.

The response seeker must address the respondent’s need prior to expecting them to accommodate his/hers.

Hot Tip: Using a mixed mode survey strategy and providing both the paper survey method for survey response with an additional option of an electronic response may address both concerns of the respondents. Some respondents may not be assured by the confidentiality of the survey process and sometimes perceive the fear of being identified through their email IDs. The paper survey option will encourage such respondents to use the paper survey instead of a no response. Conversely, some respondents may find dealing with paper and pencil difficult with the technological advancement and may prefer using the electronic version instead of a no response. Providing a mixed mode response method is likely to help respondents take the survey than to ignore it. The mixed mode methodology requires extra time, efforts and resources. However, such costs may not outweigh the benefits.

Shobha will be presenting her work on mixed methods survey research as part of the poster exhibition at AEA’s annual conference this November in San Antonio. Join us at Evaluation 2010 to connect further wish Shobha and over 1000 other presenters!

response rate, surveys

My name is Susan Kistler, the American Evaluation Association’s Executive Director, and I contribute each Saturday’s post to aea365. One hot topic for me this year is data visualization – representing data in ways that are accurate, accessible, and appealing. My very first aea365 post identified resources for those with interests in data visualization – including the classic Periodic Table of Visualization Methods. Today, I want to provide an update.

Hot Tip: Stephanie Evergreen (who gave Great Tips for Graphic Design on July 16) is working to bring together evaluators, and those working in related disciplines, who have an interest in data visualization and reporting in hopes of forming an AEA Topical Interest Group. The group would strive to build capacity and expand the knowledge base in the evaluation field in order to expand stakeholder understanding, improve interpretation, and increase use of evaluation results. As a starting point, Stephanie is building an emailing list of interested individuals and if you would like to be on the list, add a comment to this post (click through back to the website if you received this via email). She’ll also be hosting an informal meeting at Evaluation 2010!

Rad Resources: I attended a presentation in Boston given by the wonderful team at juice analytics as part of their Viva Visualization tour. The presentation was free, definitely worth the 90-minutes of my time, and gave me great ideas for improving my own reports. They’ll be coming to Washington on September 16 if you are in the area. If you aren’t in the DC area, and even if you are, you can learn from their blog – some of the best content can be found on their visitor’s guide and you can subscribe from that page as well. A couple of my favorites? Check out the post on Lightweight data exploration in Excel (under Excel Tricks) to make super-easy inline bars and  Stimulus Bill Explorer (under demos) to see an interactive tree map in action.

Rad Resource: The vizthink group on LinkedIn (you’ll need to join LinkedIn if you aren’t a member but it is free) is a great place to learn from others, post questions, and gather feedback and suggestions.

Hot Tip: Nancy Duarte, author of slide:ology, a book recommended by John Nash in his April 9 aea365 post on Creating Outstanding Presentation Slides. Nancy has a new book coming out – resonate: Present Visual Stories that Transform Audiences due on September 28. She has also created a series of short new videos on creating outstanding presentations, some of which may be found individually on Amazon, but as a set they were uploaded just this week to Vimeo for free viewing.

The above represents my own opinions and not necessarily those of AEA. This contribution is from the aea365 Tip-a-Day Alerts, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

data visualization, Graphic design, graphics, presentations

My name is Susan Kistler, and I am the Executive Director for the American Evaluation Association. I contribute each Saturday’s aea365 alert. Summer is winding down and I know that over 2500 of you are finalizing your plans for attending AEA’s annual conference this November in San Antonio. Even if you won’t be able to join us in Texas, evaluators are a traveling bunch – from visiting clients to conducting site visits to providing capacity building and training, evaluators are out and about. So, this week I’m sharing three resources that I have found to be invaluable in preparing for travel.

Hot Tip – SeatGuru: SeatGuru provides airlines seating charts for almost all flights on major airlines. You enter an airline and a flight and it shows you the type of plane and then a seat map indicating which seats are good, get mixed reviews, or are bad, including providing information about such things as ending up beside lines to the lavatories or non-reclining seats. I never choose seats on a long flight without a quick check on SeatGuru first. A few years ago, thanks to SeatGuru, my husband and I shared the only pair of seats that had extra legroom and were two-across in coach from the United States to South Korea! These seats didn’t even appear on the airline’s own online seat charts.

Hot Tip – CompareAirlineFees: This site does exactly what you would expect. It provides a chart comparing airline fees (baggage fees, change fees, unaccompanied minor fees) for US-based airlines. While many sites have such charts, and none are always accurate given the rapidly changing landscape of fees, I find the ones here to be most regularly updated and formatted for easy reading.

Hot Tip – TripAdvisor: TripAdvisor is a group-sourced travel website with reviews of hotels and destination activities as well as ideas for what to do (check out the traveler lists). The key to TripAdvisor is to be a smart consumer of the reviews – look for recommendations that have been reviewed by multiple travelers and give more weight to reviews from people who have provided lots of reviews across different places (click on a contributor’s name to see all of her or his reviews).

The above opinions are my own and do not necessarily reflect those of the American Evaluation Association. See you in San Antonio!

This contribution is from the aea365 Tip-a-Day Alerts, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

conference, flights, Travel

I am American Evaluation Association Executive Director Susan Kistler and I contribute each Saturday’s aea365 post. I have a dirty little secret – I’m obsessed with fonts! Stephanie Evergreen recently gave a great AEA Coffee Break Webinar on Graphic Design for Evaluators (she’ll also be expanding on this topic at AEA’s Annual Conference this November). Stephanie expounded on the importance of color choice and font selection to make great reports and presentations.

Rad Resources: WhatFontIs and WhattheFont both allow you to upload a picture of a font or specify a URL and they will return a best guess at the pictured font. WhatFontIs includes the option to display only free or similar free fonts and then download them on the spot – and it has hundreds in its archive (also browsable). Alternatively, WhattheFont also has a forum where font geeks will help you identify a font if you run into a dead end. Here is a great walk-through of from the MakeUseOf blog. Both of these sites are in beta, and aren’t perfect, but I’ve been impressed with the options that they’ve provided – and the sheer volume of free fonts available from WhatFontIs for free.

Stephanie encouraged her webinar attendees to use kuler to identify particular colors appropriate to a report based on those used by a client. Using tools such as WhatFontIs takes this concept one step further.

Rad Resources: This short article from Chuck Green gives examples of sets of fonts that work together to convey a mood or message. I have no eye for such things and find this guidance invaluable.

Hot Tip: Know your vocabulary when talking about fonts with a designer or using fonts in your word processing program. Here are three definitions that will help you along:

• Serifs: Serifs are the little ‘feet’ that appear on many fonts. Fonts come in two types – Serif fonts such as Times New Roman and Sans serif fonts such as Arial.
• Kerning: Kerning refers to adjusting the space between letters so that the white space is similar from letter to letter, for instance pushing a ‘A ‘and ‘W’ up close to one another producing ‘AW’ so that they actually overlap in vertical space. You can adjust kerning in Microsoft Word under the character spacing. Kerning is used in particular when creating headlines or banners.
• Proportional Typefaces: Almost all typefaces today are proportional, allotting varying amounts of horizontal space to a letter based on its shape so that an ‘l’ receives less space than an ‘m’. Old typewriters used monospace fonts.

Rad Resource: A moment of fun for the font obsessed – take a look at this comic that reflects on font choice http://ow.ly/2hf2e.

Note: These insights are my own and do not necessarily represent the opinion of the American Evaluation Association.

communication, font, presentations

My name is Lori Wingate. I am a Principal Research Associate at The Evaluation Center at Western Michigan University. Two closely related topics I return to frequently in my research, practice, and teaching are metaevaluation and the Program Evaluation Standards (Joint Committee, 1994).  Here I share some lessons learned from my recent dissertation research on the use of the Program Evaluation Standards a rating tool for metaevaluation.

The Program Evaluation Standards are a set of 30 standards organized in four domains: utility, feasibility, propriety, and accuracy. Correspondingly, they are grounded in the principles that evaluations should be useful, practical, ethical, and valid.

Because of their applicability to a broad array evaluation contexts and widespread acceptance, they are often used as criteria in metaevaluation. Although the Standards provide a useful metaevaluation framework, there are some significant challenges to their application when a metaevaluation is focused on evaluation reports, without opportunity to gather additional information about the evaluation’s conduct.

This claim is based on my personal experience in using the Standards to evaluate reports, and is strongly supported by the findings from my study of interrater agreement in metaevaluation. Although agreement was generally low across all the standards, the uncalibrated raters had the least agreement on standards in the feasibility and propriety domains, which are largely concerned with issues related to the manner in which an evaluation is carried out. With only reports in hand to judge the evaluation, raters had to infer quite a bit in order to make judgments about evaluation process.

If you’re thinking of conducting a metaevaluation in which you will use the Program Evaluation Standards as criteria and you have only evaluation reports for data, here are some tips and resources that may help make it a more valid and useful endeavor:

Hot Tip: Select only those standards on which judgments can be made based on information that is typically included in evaluation reports.

Rad Resources: Check out the Program Evaluation Standards at www.jcsee.org.  Watch for a new edition to be published this year. A review of Dan Stufflebeam’s Program Evaluation Metaevaluation Checklist will help you get started in determining which standards will be feasible for use in your metaevaluation.

Hot Tip: If you want to look at several reports produced for a by a single organization or in a single content area, spend some time developing tailored criteria for that context.

Rad Resource: ALNAP’S Quality Proforma is an instrument designed for assessing humanitarian action evaluation reports. The criteria are tailored to the domain in which the evaluations were conducted and are focused on report quality.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

metaevaluation, standards

My name is Susan Kistler. I am the American Evaluation Association Executive Director and the contributor of each Saturday’s aea365 post. This week is one of my favorites each year as it marks the start of the 2010 AEA conference ramp-up. Registration is open, the program is online, and we’re making plans for an amazing event. Here are a few ways to make the most of the conference, even if you can’t attend!

Hot Tip – Find Colleagues With Similar Interests: The AEA Conference program includes information about over 600 sessions and hundreds of individual presentations. Because it is fully searchable, the online program is not only a great resource for planning conference attendance, but also for identifying and connecting with others doing work in your area of interest.

Hot Tip – Make the Most of Collateral Materials: This year, we’ll have over 100 conference presenters making contributions to the aea365 Tip-a-Day Alerts (be sure to subscribe if you have not already), and conference presenters will be encouraged to upload materials into the AEA Public eLibrary. Even those who cannot attend, may receive great advice via the alerts from conference presenters, and all materials in the eLibrary are freely downloadable.

Hot Tip – Attend Workshops: There are over 50 professional development workshops that precede or follow the conference. These are affordable, investigate both foundational and cutting edge methods and issues, and many are taught by some of the most revered leaders in the field. You can attend these whether or not you attend the conference itself. Because there are three days of workshops in advance of the conference, you can make a great customized training institute by selecting the best 1, 2, or 3 days of workshops that suites your needs.

Hot Tip – Attend the Conference! AEA’s Annual Conference will be held November 10-13 in San Antonio, Texas. While we have great ways for those who cannot attend to benefit still from the knowledge exchange, nothing substitutes for experiencing Evaluation 2010 face-to-face. You’ll have the opportunity to build your network, engage in great sessions, learn in a supportive environment, and explore a city steeped in history. Visit the Evaluation 2010 website to learn more.

No tags

Hi, I’m Dawn Hanson Smart, Senior Associate at Clegg & Associates, Inc. in Seattle, a consulting firm focused on planning and evaluation with nonprofits, state and local governments, tribal organizations, and foundations. I’ve been an evaluator for more than 25 years, with an eclectic mix of clients and projects. I think it is this diversity of fields and topics that keeps me interested and excited about my work. It also leads me to explore a wide range of reading material. I recently finished The Immortal Life of Henrietta Lacks by Rebecca Skloot, the story of the woman whose cells became the first to be grown in a laboratory and the many related stories of how it came about and the impact on her family and on research science.

Lesson Learned: Beyond the admiration we may feel for their work, these kinds of authors can stimulate our thinking, bringing new perspectives to the way we approach our own. Rebecca Skloot’s persistence over more than ten years to gather the data she needed, her diligence in the face of barriers, and her unfailing respect and caring for the people she worked with inspired me. The book made me think about how I might adjust my approach to (and my attitude about) a difficult evaluation I was conducting. It also made me remember the encouragement and enlightenment provided by books in the past. So I challenge you to broaden your reading habits and explore.

Rad Resource: Great places to get book ideas and see reviews include the New York Times Sunday Book Review and the New Yorker’s, Books Briefly Noted.

Rad Resource: TED is a nonprofit “devoted to Ideas Worth Spreading.” TED’s annual conference brings together speakers about science, business, the arts and global issues. Over the conference’s four days, 50 speakers each get an 18-minute time slot to share their knowledge and ideas and to inspire the audience with their creativity and passion for their work. You can see video clips of conference speakers at http://www.ted.com/ and identify individuals and topics you’d like to know more about.

Hot Tip: Book clubs are terrific places to learn about books you might not otherwise select for yourself. If you’re not a part of a club, join one. If you’re in one that’s beginning to feel stale, seek out a new one that includes nonfiction, fiction, history, biography, and poetry books. Or join a club online. The Salon Reading Club offers a stimulating community where you can get together with others and discuss books. Or, try the New Yorker’s Book Club.

So, explore — read outside your field something in an area you’ve never even considered of interest. You might be surprised!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

book, Data, Reading, TED

I’m Leslie Goodyear, and I’m a Program Officer at the National Science Foundation, in the Division of Research on Learning in Formal and Informal Settings (DRL). The programs in DRL include: Informal Science Education (ISE), Discovery Research K-12 (DR K-12), Research and Evaluation on Education in Science and Engineering (REESE), and Innovative Technology Experiences for Students and Teachers (ITEST). I have a hot tip about how to become a proposal reviewer for NSF/DRL.

NSF’s proposal merit review process generally includes review by outside experts. For DRL, experts in Science, Technology, Engineering and Mathematics (STEM) education, research methods, learning sciences, evaluation, and other areas are typically brought together in panels. They discuss the relative merits of the proposals and offer their best thinking to NSF programs officers based on two primary review criteria: intellectual merit and broader impacts. Advice by reviewers and panels is critical to informing program officers, who make the recommendations for awards.

As a panel reviewer, you’ll read about 15 to 20 proposals (each is about 15 pages long); write reviews for about six to eight proposals; join about ten to 12 colleagues in a two-day review panel in Arlington, Va., home to NSF; discuss the proposals and the reviews; and rate the proposals as a priority for funding. Reviewers who travel to NSF are paid a stipend for the days they serve on the panel and their travel is covered by NSF; ad hoc reviewers, who normally review just one or two proposals without serving on a panel, are not paid. In addition to providing a valuable service to the NSF and the field, you’ll learn a lot about what makes a good proposal and how the review process at NSF works. Most people who participate think it’s a great professional development opportunity.

Because DRL programs require project evaluation, the proposals submitted include evaluation plans. Thus, DRL always needs experienced, competent evaluation professionals to gauge the quality of these plans. We primarily look for evaluators who have experience conducting evaluations of STEM education programs. We also look for evaluators with strong methodological training, experience with formal or informal educational settings (in-school or out-of-school), expertise in evaluating research, and practical expertise in evaluating community programs.

Hot tip: If you’d like to be considered for serving as a proposal reviewer, first go to the NSF website (below) and learn about our programs by reading the program solicitations. Then send your CV and a cover letter with a bit about yourself, your expertise and experience, and the program(s) for which you’d like to serve as a reviewer to me, Leslie Goodyear, lgoodyea@nsf.gov. I will then forward them to the appropriate cluster within the division.

NSF DRL Website: http://nsf.gov/div/index.jsp?div=DRL

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

National Science Foundation, NSF, Peer Review

I am Martha (Marty) Henry, founder and President of M.A. Henry Consulting, LLC in St. Louis.  Today I will be sharing a story and some hot tips about data confidentiality and data ownership.

My firm was evaluating a federally funded project at a university under IRB oversight with approved consents for participants.  The consents stated that only evaluators (we were named in the consent as evaluators) would have access to the key that linked participant codes to participant names.  Two-and-a-half years into the five-year project, the program staff required that evaluators turn over all data with identifiers for internal uses.

Discussions with the university IRB confirmed our position to provide coded data to the project without identifiers as conforming to the consents sent out on university letterhead and signed by the PI. The university program staff, none of whom had taken the CITI training in human subjects, referred to the university contract language that said that all data were the property of the university. Their interpretation was that the data should be turned over with participant names.

We maintained our position and were fired as evaluators.  The university then demanded the data as we were no longer the evaluators for the project.  After retaining legal council and discussions with IRB experts and federal oversight staff, we reiterated that we would turn over coded data at any point but not identifiable data.  Thirteen months later the university agreed to receive the coded data.

Hot Tip:  Before signing any contract that references data ownership, obtain interpretation from your legal counsel regarding congruity of the language as stated in the contract and in the participant consents.  If there is any possibility for dual interpretation, add an amendment to the contract clarifying release of confidential, identifiable data and which staff may have access to it.

Hot Tip:  Before agreeing to evaluate any project, assure that all project staff have passed CITI.  CITI is a series of training and education modules about the history, ethics, and practices necessary for protecting human participants, which generally include anyone participating in a study. The CITI training modules and tests are available at https://www.citiprogram.org/Default.asp?

Hot Tip:  If anything seems amiss in your beginning discussions with clients, trust your gut!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

CITI, confidentiality, data ownership, Database management, ethics

I’m Linda Lee, partner in a Canadian social research and program evaluation company. We work in the public and not-for-profit sectors often evaluating programs that concern children and youth. Both in our work in Canada and internationally, we are frequently working in communities that have been marginalized, such as First Nations communities in Canada or Roma communities in East Central and South-eastern Europe.

As we worked with children and in these diverse communities, we recognized that many of our standard methods for data collection seemed artificial or inappropriate. Conventional methods did not always seem to be giving voice to participants or eliciting the ‘real story.’ So, along with more traditional methods (e.g., surveys, interviews) we began incorporating visual methods into our evaluations. Some examples of visual methods are:

• Drawings (created by the participant in response to a question or prompt)
• Graphic representations (e.g., webs, mind-maps, graphic organizers)
• Photo-interviews (people respond to photographs taken by others, or take their own photographs to answer a question or tell a story)

The list can easily be expanded to include other visual media. The methods listed above are simple and low cost. It is reasonably easy to incorporate them into interview situations.

Lessons Learned: We found there were many advantages to using visual methods. Such methods:

• can be used at any stage of applied research or evaluation,
• provide a connection between psychological and physical realities,
• allow for the combination of visual and virtual language,
• are useful with respondents who have emerging literacy levels in the mainstream language,
• help to build trust and relationships,
• produce unpredictable information which may help get to the root issues,
• promote longer, more detailed interviews,
• support triangulation (in combination with other methods),
• support collaborative/participatory research,
• are liked better than conventional interviews by many participants,
• support engagement of groups whose voices are not usually well heard.

Hot Tips and Recommendations:

• Try using one of these methods in an evaluation, but start by coupling it with a more standard method. Many of these methods do not work well in isolation. Plus, because they are less conventional, they are not immediately credible to some audiences.

• You can use them pre-post!

• Drawing often works better with children than with adults who can sometimes feel self-conscious.

• Graphic representations are familiar to children and youth as they are used by in schools. Adults may need a little more direction if they are asked to create a web or mind-map.

• Having respondents take photographs to tell their story can be very powerful, but needs to be part of a longer term process – not one shot data collection.

• Be inventive and have some fun!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

visual facilitation