Hello! I am Carey Tisdal, Director of Tisdal Consulting in St. Louis, Missouri. I work with people who develop informal learning experiences for museum exhibitions, museum programs, documentary films, and media-based projects. Many of my projects include websites as one element of a learning system. I used the Building Informal Science Education (BISE) project as an opportunity to develop a framework to focus studies involving websites. This experience helped me improve my own practice by analyzing other evaluators’ work as well as connecting to key concepts in the website evaluation literature. I hope you find it useful, too!
I developed my website evaluation framework by analyzing 22 reports from the BISE database that were coded as “website” evaluands (i.e. the entity being evaluated). The overarching method I used to analyze the reports was Glaser & Strauss’ Grounded Theory. I then connected concepts in the program theory to literature about website evaluation. The resulting website evaluation framework uses high-level program theory to guide the identification of focus areas and questions to structure website evaluations. As illustrated in the graphic below, I organized seven of the major areas of consideration as a set of sequential, necessary steps influencing User Impacts and System Effectiveness. Read my whitepaper, “Websites: A guiding framework for focusing website evaluations,” to learn more!
Lessons Learned:
- Some of the evaluations I reviewed focused on appeal (content, visuals, or forms of engagement), which is certainly a very important aspect of website evaluation. Yet, when connecting the focus areas, I realized that without testing usability, as well as appeal, it is not possible to draw strong conclusions about how audience impact is or is not accomplished.
- Evaluating the system effectiveness of a website is essential in multiplatform projects. Awareness and access play important roles in whether or not users of other parts of an informal education system (e.g. an exhibition, program, or film) even get to the website, or, in turn, if website viewers see a film or attend an exhibition.
- In my own work, I’ve found that this website framework helps project teams and website designers to clarify what they really need to know.
Rad Resources:
- The U.S. Department of Health and Human Services offers an amazing set of resources to get you started in usability testing for websites. This site has been updated since I did my research and is now even better!
- The BISE database and the website org provide access to a wide range of evaluation reports. When I need to look at how colleagues approached evaluation designs, they are my first stops!
The American Evaluation Association is celebrating Building Informal Science Education (BISE) project week. The contributions all this week to aea365 come from members of the BISE project team. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.