Hi! This is Amy Grack Nelson, Evaluation and Research Manager, and Zdanna King, Assistant Evaluation and Research Manager, from the Science Museum of Minnesota. If you are like us, you may have shared an evaluation report with other evaluators on websites such as AEA’s eLibrary or informalscience.org. Even though opportunities to share reports online are increasing, the evaluation field lacks guidance on what to include in evaluation reports meant for an evaluator audience. If the evaluation field wants to learn from evaluation reports posted to online repositories, how can evaluators help to ensure the reports they share are useful to this audience? As part of the Building Informal Science Education (BISE) project, we explored this question through the analysis of 520 evaluation reports uploaded to informalscience.org. The BISE team created an extensive coding framework to align with features of evaluation reports and evaluators’ needs. We then used the framework to identify how often elements were included or lacking in evaluation reports.
Lessons Learned: Our analysis brought to light where reports in the field of informal education evaluation may already meet the needs of an evaluator audience and where reports are lacking. To help maximize learning and use across the evaluation community, we developed guiding questions evaluators can ask themselves as they prepare a report to share with other evaluators.
- Have I described the project setting in a way that others will be able to clearly understand the context of the project being evaluated?
- Is the subject area of the project or evaluand adequately described?
- Have I identified the type of evaluation (formative, summative, etc.)?
- Is the purpose of the evaluation clear?
- If I used evaluation questions as part of my evaluation process, have I included them in the report?
- Have I described the data collection methods?
- If possible, can I include data collection instruments in the report?
- Do I provide sufficient information about the sample characteristics? If I used general terms such as “visitors,” “general public,” or “users,” do I define what ages of individuals are included in that sample?
- Have I reported sample size for each of my data collection methods?
- If I report statistically significant findings, have I noted the statistical test(s) used? Do I only use the word “significant” if referring to statistically significant findings?
- If I provided recommendations to the client, did I include them in the report?
Rad Resources:
- Read our full paper, Reporting with an Evaluator Audience in Mind, which describes the study findings that led to our guiding questions.
- A great checklist we used to inform the development of our coding framework and guiding questions is Miron’s Evaluation Report Checklist.
The American Evaluation Association is celebrating Building Informal Science Education (BISE) project week. The contributions all this week to aea365 come from members of the BISE project team. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.