Scott Vollmer

Hi, I’m Scott Vollmer, VP of STEM Learning at Great Lakes Science Center (GLSC) in Cleveland, Ohio.  I am passionate about my work and find that evaluation data is especially useful when trying to express my passion objectively.

Since July 2017, we have been using COVES (the Collaboration for Ongoing Visitor Experience Studies) to collect and analyze audience-level data to enhance the guest experience. We have collected 858 surveys through the first year. One of the measurements included in COVES’ five-minute voluntary exit survey is the widely-used loyalty indicator known as Net Promoter Score® (NPS). Further, COVES collects data based on motivation, visitation trends, and guest demographics.

The data-parsing tools available through COVES are extensive and we stick to our five-year Strategic Plan while interpreting our results. Goals three and five of our Strategic Plan focus on investing in the guest experience of our target audience, and encourage return/frequent visits.

Sticking to the Strategic Plan ensures that we maximize our Return On Investment (ROI) when making improvements.  The clearest example of this so far can be seen when looking at the NPS attached to our target demographic experience. Our core audience is families with children between the ages of two and twelve. When we look at only that demographic, our NPS is in the low 70s. This is exactly where we want to be, overall. When looking at adults without children, our NPS ranges from an abysmal 16 and only makes it as high as 57.5. This proves that we will see a greater ROI when focusing on our core audience. With COVES, we also have the luxury of diving deeper into exhibit-specific experiences. We have learned that our time and money is best spent on guest/staff interactions (exhibit interpretation, big shows, cart activities, etc.). Finally, we are able to track ROI results from our multi-day, Value-Added Programs. These Programs require doubling engagement staff and significant investments in immersive environments. Once these events are complete, we are able to see if there was improvement in the guest experience, allowing us to choose whether or not to make that investment again in the future.

Hot Tip:

COVES is also a great tool for reporting out to our Board of Directors.  We have created custom online dashboards using COVES data that are presented alongside more traditional Profit and Loss, Daily Gate Attendance, and Revenue reports. Rather than forcing arbitrary engagement goals, we use national aggregate COVES data to set benchmarks that our Board can easily understand. The ability to segment national data from similar medium-sized institutions allows us to have an apples-to-apples comparison.

The Great Lakes Science Center plans to continue using COVES to drive many of our decisions regarding exhibits, live interactions, events, traffic flow, and overall guest experience. We are certain that we can continue to build on past successes and spot negative trends before it is too late to make a difference.

We’re looking forward to the fall and the Evaluation 2018 conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

lawg · Visitor Studies

My name is Elizabeth Bolander, Director of Audience Insights and Services at the Cleveland Museum of Art (CMA) and President-Elect of the Visitor Studies Association. We do a variety of research and evaluation activities at CMA, all with the goal of learning about the needs of our various constituencies and communities. This work is particularly important in light of our new strategic plan, which recommits us to our mission of “for the benefit of all the people, forever.”

Over the last few years, we’ve completed several studies focusing on the barriers to entry among non-visitors. This has ranged from community-based panels where we worked with local libraries to host and recruit residents for discussions to focus groups with infrequent and non-visitors that met certain demographic criteria (i.e. young professionals, parents with children under age 8 in the household, etc.).

Lessons Learned:

Establishing mutually beneficial partnerships is critical to success

When we wanted to reach residents in traditionally underserved communities by the museum, we knew we needed a partner that could help establish trust and generate participant interest. Yet, we also wanted to ensure the partner was getting something out of the relationship as well. When we worked with local library branch managers, they helped spread the word about the open discussion groups and made sure we got our target respondents for the study, while we made sure they also received our findings.

Prepare internal stakeholders

Many of the key insights we gleamed from these non-visitor studies were not always the easiest to hear because they often challenged the status-quo and internally held assumptions. Laying the groundwork in advance and encouraging ongoing conversations after the formal presentation helped make the findings easier to digest and immediately actionable.

Research as a community engagement tool

Going out into the communities and listening to their thoughts and opinions helped create additional in-roads in these neighborhoods. Some community members felt more apt to attend CMA programming because we took the time to listen to them and engage.

Rad Resources:

Even if you think you’re not an “art museum person” like some of our respondents, I hope many of you will come see us while you’re attending AEA in Cleveland this fall. In addition to the Cleveland Museum of Art, the University Circle neighborhood  is home to the Cleveland Museum of Natural History, Cleveland Botanical Garden, Cleveland History Center, MOCA Cleveland, and the Cleveland Orchestra, as well as being adjacent to Little Italy. The RTA Health Line bus departs from Tower City as does the Rapid train, where you can take the red line to the Cedar-University Circle or Little Italy-University Circle stops. The free CircleLink bus operates year-round throughout the neighborhood, or you can take a stroll through the beautiful parks and Case Western Reserve University campus.

We’re looking forward to the fall and the Evaluation 2018 conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

Link to www.universitycircle.org

lawg

Hi, my name is Keisha Farmer-Smith from Become, better known as Dr. Kei-Kei! I received my nickname from young people in an afterschool program while conducting evaluation and outcomes work. It is an honor to learn from children and help create quality youth development programs. Artwork is one way to creatively explore program strengths, opportunities, and challenges.

Recently, I had the opportunity to use artwork in evaluating an innovative program, Chicago Pre-College Science and Engineering (ChiS&E). In ChiS&E, “little engineers” aged 6-9 explore their community and have fun! The program’s mission is to increase the number of historically underrepresented African American and Latino students who are motivated and academically prepared to pursue careers in science, technology, engineering and mathematics (STEM).  As a part of a longitudinal program evaluation, ChiS&E youth participated in interviews with an art component.  Little engineers were invited to draw pictures of their ChiS&E experiences:

1st grade drawings- students demonstrate a strong understanding of experiment steps and the chemical engineers’ tasks when combining components to get a reaction as well as enthusiasm about lab work.  The figures in the pictures are smiling and working.

Elijah: We put the vinegar and baking soda in the water bottle, we went outside and put the cork on it the cork exploded off the water bottle.

Evaluator: Awesome! And that’s what chemical engineers do? They do things like that?

Elijah: (nods head)

Evaluator: And what about you, Paola, what did you draw?

Paola: I was trying to put the hand up here to hold it and the bottle was bubbling, and the bubbles that had air on it was putting the air inside the balloon.

Lessons Learned: When evaluating youth programming, art-based activities are an exceptional way to learn because:

• Drawing artwork together builds rapport and engages program participants. Young people may feel uncomfortable talking to unknown evaluators/adults. Art-based activities are a fun, nonthreatening way to break the ice.

• Art is a great communication tool. When young people explain their art, relevant and meaningful information about programming is shared. During this process, it is incredibly important for adult evaluators to actively listen as youth share the intent and meaning behind their artwork.

• Art can be a strength-based tool. When young people make art it affirms their vision, voice, and opinion. As the artist, the young person has power to explore topics that are important to them.

• Using art as an evaluation tool helps evaluators collect rich, interesting and valuable information. Sometimes, it is easier to express emotions and experiences while drawing. Art-based evaluation should always be accompanied by a listening or discussion session.

• Using art as an evaluation tool is not difficult. From stick figures to complicated images, art is fun for everyone! The artist’s skill level doesn’t matter because the story behind the art can still be shared.

Rad Resources:

Americans for the Arts and Boston Youth Arts Evaluation Project demonstrate two of the many resources for using art in youth program evaluation.

The American Evaluation Association is celebrating Become: Community Engagement and Social Change week. The contributions all this week to aea365 come from authors associated with Become. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Greetings, I am June Gothberg, Ph.D. from Western Michigan University, Chair of the Disabilities and Underrepresented Populations TIG and co-author of the Universal Design for Evaluation Checklist (4^th ed.).   Historically, our TIG has been a ‘working’ TIG, working collaboratively with AEA and the field to build capacity for accessible and inclusive evaluation.  Several terms tend to describe our philosophy – inclusive, accessible, perceptible, voice, empowered, equitable, representative, to name a few.  As we end our week, I’d like to share major themes that have emerged over my three terms in TIG leadership.

Lessons Learned

• Representation in evaluation should mirror representation in the program. Oftentimes, this can be overlooked in evaluation reports.  This is an example from a community housing evaluation.  The data overrepresented some groups and underrepresented others.

• Avoid using TDMs.
  • T = tokenism or giving participants a voice in evaluation efforts but little to no choice about the subject, style of communication, or any say in the organization.
  • D = decoration or asking participants to take part in evaluation efforts with little to no explanation of the reason for their involvement or its use.
  • M = manipulation or manipulating participants to participate in evaluation efforts. One example was presented in 2010 where food stamp recipients were required to answer surveys or they were ineligible to continue receiving assistance.  The surveys included identifying information.
• Don’t assume you know the backgrounds, cultures, abilities, and experiences of your stakeholders and participants. If you plan for all, all will benefit.
  • Embed the principals of Universal Design whenever and wherever possible.
  • Utilize trauma-informed practice.
• Increase authentic participation, voice, recommendations, and decision-making by engaginge all types and levels of stakeholders in evaluation planning efforts. The IDEA Partnership depth of engagement framework for program planning and evaluation has been adopted in state government planning efforts across the United States.

• Disaggregating data helps uncover and eliminate inequities. This example is data from Detroit Public Schools (DPS).  DPS is in the news often and cited as having dismal outcomes.  If we were to compare state data with DPS, does it really look dismal?

Disaggregating by one level would uncover some inequities, but disaggregating by two levels shows areas that can and should be addressed.

• Tailor your reporting and dissemination efforts to be engaging and accessible by all stakeholders. The following video from the Denver Art Museum shows both an engaging and accessible (option for closed captioning) presentation of the evaluation findings.

We hope you’ve enjoyed this week of aea365 hosted by the DUP TIG.  We’d love to have you join us at AEA 2017 and throughout the year.

The American Evaluation Association is hosting the Disabilities and Underrepresented Populations TIG (DUP) Week. The contributions all week are focused on engaging DUP in your evaluation efforts. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

dropout · leadership · museum studies · school data · Universal Design for Evaluation

My name is Diego Escobar and I am Director of Institutional Development at Jalisco’s Ministry of Culture. We recently hosted an event that was part of CLEAR-LA’s Evaluation Week Mexico 2016. We were very excited to be part of the program.

We took the opportunity to ignite discussion and bring together experts from what sometimes appear to be isolated fields, such as arts promotion and public administration and evaluation.

The event produced consensus around the following issues (among others):

• Cultural programs deal with issues that are specific to the field and pose endemic “evaluation challenges.” For example, grant programs may eventually need to establish and compare the artistic merit of different projects funded in any given year, which will hardly be achieved without debate.
• Nonetheless, much can be learned from evaluation in other fields and government sectors. For this to happen it is crucial to identify cultural programs with similar interventions to those of programs seeking non-artistic goals (i.e. anything from improving employment to promoting hygiene habits). Translating knowledge is vital.

Lesson Learned: Methodology! In Latin America (and beyond) the arts and the artistic components of community programs have extraordinary powers. This raises the stakes for evaluation. How do we know if there is a relationship between arts engagement in youth and criminal behaviours in adulthood? A report by Giving Evidence and UCL’s Institute of Education explores the relationship between short term and long term outcomes produced by outdoor adventure programs in the UK. The arts sector can use this approach to research and strengthen its claims regarding the relationship between effective programs in the short term and likely outcomes in the long-term.

Hot tip: Arts organizations don’t usually have the budget to send professionals or staff around to evaluation conferences or seminars. Although their participation in said events is growing, it might be a good idea to find ways for people and organizations to engage easily and without travel expenses. Using Facebook’s live streaming, we were able to boost our audience and reach people that made interesting contributions to the discussion.

Rad Resource: If you are in charge of evaluating (or managing!) a program it is useful to see if other organizations have commissioned evaluations for programs similar to the one you are interested in. Although it is often said that there is very little going on regarding arts and evaluation, I’ve found it very useful to check the International Federation of Arts Councils and Culture Agencies’ publications registry.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hi! I’m Sarah Cohn, Association Manager for the Visitor Studies Association (VSA). VSA and I are a part of the Building Informal Science Education project (BISE). In this post, I’ll share information about VSA and how the Association supported the BISE project.

Lessons Learned: VSA served as a platform for the BISE project to engage with a community of evaluators and researchers throughout the field of informal learning, united around a set of common goals:

• to ensure the project’s work was grounded in the experiences of these audiences,
• to ensure that the resulting resources were useful and relevant to the wide range of evaluators that might use them.

At VSA annual meetings, the BISE project team sought input regarding:

• the development of the BISE Coding Framework,
• the direction of the project, and
• the findings from the synthesis papers.

This sort of member-checking is important in qualitative research to ensure that the findings are truly reflective of the audience’s perspectives or information. Evaluators can be just as tricky and diverse in their ideas, needs, and opinions as any other audience! So what did we learn from this process?

• When conducting research on evaluation, provide multiple venues and points at which evaluators can reflect on the study’s data, ideas, and findings. Find different check-in points over the course of a year or the life of the project; offer different modes for engagement, be they digital, in-person, asynchronously, etc.
• Be as specific as possible in your requests for feedback. We are reflective, by nature, so your fellow evaluators will provide feedback on every aspect of a project if you let them!

Rad Resources: The Visitor Studies Association is a global network of professionals dedicated to understanding and enhancing learning experiences in informal settings wherever they may occur—in museums, zoos, parks, visitor centers, historic sites, and the natural world—through research, evaluation and dialogue. VSA’s membership and governance encompass those who design, develop, facilitate, and study learning experiences. We offer a number of resources for evaluators to learn more about evaluation in informal settings.

• An annual conference every summer that brings over 200 professionals together to talk about new advances in the field, current projects, and major issues they are facing.
• The Visitor Studies journal, which is a bi-annual, peer-reviewed journal that publishes high-quality articles focusing on research and evaluation in informal learning environments, reflections on the field, research methodologies, and theoretical perspectives. The journal covers subjects related to museums and learning in out-of-school settings.
• Online webinars, produced in partnership with other museum-related associations, such as the Association for Science-Technology Centers.
• Regional meet-ups, workshops, and an active listserv.

The American Evaluation Association is celebrating Building Informal Science Education (BISE) project week. The contributions all this week to aea365 come from members of the BISE project team. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Visitor Studies

Hello! I am Rebecca Teasdale, a doctoral student in Educational Psychology specializing in evaluation methodology at the University of Illinois at Urbana-Champaign. I’m also a librarian and have served as an administrator and science librarian in public libraries. My current work focuses on the evaluation of interest-driven learning related to science, technology, engineering and math (STEM) that takes place in public libraries and other informal learning settings.

I first became involved with the Building Informal Science Education (BISE) project as an intern at the Science Museum of Minnesota while I was pursuing a certificate in evaluation studies at the University of Minnesota. (See my blog post, “Measuring behavioral outcomes using follow-up methods,” to learn more). Now, I’m using the BISE database to support my research agenda at Illinois by identifying methods for evaluating the outcomes of public library STEM programming.

Evaluation practice is just getting started in the public library context, so few librarians are familiar with evaluation methods measuring mid- and long-term outcomes of informal science education (ISE) projects. I used the BISE reports to provide a window into understanding (a) the types of outcomes that ISE evaluators study, (b) the designs, methods and tools that they use, and (c) the implications for evaluating the outcomes of STEM programs in public libraries.

Lessons Learned:

• I’ve found little standardization among the evaluation reports in the BISE database. Therefore, rather than provide a single model for libraries to replicate or adapt, the BISE database offers a rich assortment of study designs and data collection methods to consider.
• Just 17% of the reports in the BISE database included the follow-up data collection necessary to examine mid- and long-term outcomes. In particular, library evaluators should ensure that we design studies that examine these effects as well as more immediate outcomes.
• Collecting follow-up data can be challenging in informal learning settings because participation is voluntary, participants are frequently anonymous, and engagement is often short-term or irregular. The reports in the BISE database offer a number of strategies that library evaluators can employ to collect follow-up data.
• All five impact categories from the National Science Foundation-funded Framework for Evaluating Impacts of Informal Science Education Projects are represented in the BISE database. I’m currently working to identify some of the methods and designs for each impact category that may be adapted for the library context. These impact categories include:
  • awareness, knowledge or understanding
  • engagement or interest
  • attitude
  • behavior
  • skills

 Rad Resource:

• I encourage you to check out the BISE project to inform evaluation practice in your area of focus and to learn from the wide variety of designs, methods, and measures used in ISE evaluation.

The American Evaluation Association is celebrating Building Informal Science Education (BISE) project week. The contributions all this week to aea365 come from members of the BISE project team. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hi! This is Amy Grack Nelson, Evaluation and Research Manager, and Zdanna King, Assistant Evaluation and Research Manager, from the Science Museum of Minnesota. If you are like us, you may have shared an evaluation report with other evaluators on websites such as AEA’s eLibrary or informalscience.org. Even though opportunities to share reports online are increasing, the evaluation field lacks guidance on what to include in evaluation reports meant for an evaluator audience. If the evaluation field wants to learn from evaluation reports posted to online repositories, how can evaluators help to ensure the reports they share are useful to this audience? As part of the Building Informal Science Education (BISE) project, we explored this question through the analysis of 520 evaluation reports uploaded to informalscience.org. The BISE team created an extensive coding framework to align with features of evaluation reports and evaluators’ needs. We then used the framework to identify how often elements were included or lacking in evaluation reports.

Lessons Learned: Our analysis brought to light where reports in the field of informal education evaluation may already meet the needs of an evaluator audience and where reports are lacking. To help maximize learning and use across the evaluation community, we developed guiding questions evaluators can ask themselves as they prepare a report to share with other evaluators.

1. Have I described the project setting in a way that others will be able to clearly understand the context of the project being evaluated?
2. Is the subject area of the project or evaluand adequately described?
3. Have I identified the type of evaluation (formative, summative, etc.)?
4. Is the purpose of the evaluation clear?
5. If I used evaluation questions as part of my evaluation process, have I included them in the report?
6. Have I described the data collection methods?
7. If possible, can I include data collection instruments in the report?
8. Do I provide sufficient information about the sample characteristics? If I used general terms such as “visitors,” “general public,” or “users,” do I define what ages of individuals are included in that sample?
9. Have I reported sample size for each of my data collection methods?
10. If I report statistically significant findings, have I noted the statistical test(s) used? Do I only use the word “significant” if referring to statistically significant findings?
11. If I provided recommendations to the client, did I include them in the report?

Rad Resources:

• Read our full paper, Reporting with an Evaluator Audience in Mind, which describes the study findings that led to our guiding questions.
• A great checklist we used to inform the development of our coding framework and guiding questions is Miron’s Evaluation Report Checklist.

The American Evaluation Association is celebrating Building Informal Science Education (BISE) project week. The contributions all this week to aea365 come from members of the BISE project team. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! I am Carey Tisdal, Director of Tisdal Consulting in St. Louis, Missouri. I work with people who develop informal learning experiences for museum exhibitions, museum programs, documentary films, and media-based projects. Many of my projects include websites as one element of a learning system. I used the Building Informal Science Education (BISE) project as an opportunity to develop a framework to focus studies involving websites. This experience helped me improve my own practice by analyzing other evaluators’ work as well as connecting to key concepts in the website evaluation literature. I hope you find it useful, too!

I developed my website evaluation framework by analyzing 22 reports from the BISE database that were coded as “website” evaluands (i.e. the entity being evaluated). The overarching method I used to analyze the reports was Glaser & Strauss’ Grounded Theory. I then connected concepts in the program theory to literature about website evaluation. The resulting website evaluation framework uses high-level program theory to guide the identification of focus areas and questions to structure website evaluations. As illustrated in the graphic below, I organized seven of the major areas of consideration as a set of sequential, necessary steps influencing User Impacts and System Effectiveness. Read my whitepaper, “Websites: A guiding framework for focusing website evaluations,” to learn more!

Lessons Learned:

• Some of the evaluations I reviewed focused on appeal (content, visuals, or forms of engagement), which is certainly a very important aspect of website evaluation. Yet, when connecting the focus areas, I realized that without testing usability, as well as appeal, it is not possible to draw strong conclusions about how audience impact is or is not accomplished.
• Evaluating the system effectiveness of a website is essential in multiplatform projects. Awareness and access play important roles in whether or not users of other parts of an informal education system (e.g. an exhibition, program, or film) even get to the website, or, in turn, if website viewers see a film or attend an exhibition.
• In my own work, I’ve found that this website framework helps project teams and website designers to clarify what they really need to know.

Rad Resources:

• The U.S. Department of Health and Human Services offers an amazing set of resources to get you started in usability testing for websites. This site has been updated since I did my research and is now even better!
• The BISE database and the website org provide access to a wide range of evaluation reports. When I need to look at how colleagues approached evaluation designs, they are my first stops!

The American Evaluation Association is celebrating Building Informal Science Education (BISE) project week. The contributions all this week to aea365 come from members of the BISE project team. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

usability testing · website

Greetings! I’m Beverly Serrell, museum exhibition consultant, evaluator, and developer with Serrell & Associates in Chicago, Illinois. As a practitioner, I am very interested in finding helpful information to improve my practice in the planning, development, and assessment of exhibits. When the Building Informal Science Education (BISE) project invited me to look at their database and investigate a question of my choice, I was most curious about recommendations in summative evaluation reports. Did the advice, (e.g. recommendations or suggestions for improvements) compare to mine? Were there trends that could be shared and applied?

I started my report by looking at 50 summative evaluation studies in the BISE database that were coded as including “recommendations.” Further sorting brought the list down to 38—with a diverse selection of science disciplines, (e.g., botany, zoology, astronomy, biology, ecology, geology, and health sciences).

Lesson Learned: Orientation was often the single biggest challenge to get right in exhibitions. Using a bottom-up method of review, the issue that emerged as most common included the need for better orientation within an exhibition. Recommendations for improvements to orientation came from problems related to the various physical and psychological needs of museum visitors. Two other suggestions were closely tied to orientation: more clarity in conceptual communication and better delineation of exhibit boundaries. These recommendations and more are discussed and examples are given in my full report, “A Review of Recommendations in Exhibition Summative Evaluation Reports.”

Hot Tip: Criticism is about the work, and the work can always be improved. Whether to include a section on recommendation in an exhibitions summative evaluation is somewhat controversial. Some evaluators think that it is the client’s job––not the evaluators––to interpret the data, and that making recommendations for improvements can cast a negative light on the institution and hurt its reputation with funders. It is important for evaluators to make sure at the outset of a project that the client is eager to hear the thoughts of an experienced evaluator.

My advice for making recommendations in summative evaluation reports is to go ahead and make them. Without couching them in meek tones, be specific and give the context and evidence for why the recommendation is being made. Evaluation is recognized today as a valuable part of the process; it’s no longer us (evaluators) against them (designers, curators, etc.).

My favorite example of an exhibition report with numerous indicators of success and a balanced offering of practical suggestions for improvements is Sue Allen’s 2007 summative evaluation of “Secrets of Circles” at the San Jose Children’s Museum.

The American Evaluation Association is celebrating Building Informal Science Education (BISE) project week. The contributions all this week to aea365 come from members of the BISE project team. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

recommendations · summative