My name is Randahl Kirkendall. I work part-time as an Evaluator with Ellen Iverson, Director of Evaluation, for the Science Education Resource Center (SERC) at Carleton College, which works to improve education through projects that support educators.  Our work is funded primarily through NSF grants.  SERC has expertise in geoscience education, workshop leadership, website development and program and website evaluation.

A primary aim of SERC is to help faculty adopt evidence-based teaching behaviors that will enhance student learning. In evaluating the websites at SERC, our interest is in the role of website use in faculty professional development. We use a variety of web analytic tools such as Google Analytics, server-based website statistics, and web page visit logs in combination with data from surveys, interviews, focus groups, and observations to get as complete a picture as possible for how faculty use  websites and the impact that their use has on teaching behavior.

Lesson Learned: One of the things we have learned from user interviews is that people generally have poor recall of how they found a website and used it. While they can explain why they go to a website (motivation), they have difficulty recalling at what section of the website they started, what pages they viewed, and the search strategy they used. Website use analytics and web server logs of individual visits provide a richer picture of user behavior and interests via records of the actual pages that they visited.

Lesson Learned: The SERC websites often don’t work in isolation. Our survey of 2,000+ faculty found that a significant number of users were using the websites to compliment other professional development activities such as attending workshops, exchanging ideas with colleagues, or reviewing literature. Thus, it has been prudent that we collect data on these other possible influences on their teaching behavior.

Cool Trick: We sequence or build evaluations incrementally, partially basing data collection and/or analyses on findings from other data collection methods. For example, we use the findings from user interviews to describe predominant motivations for using a website and any changes in behavior (such as teaching practice changes) that users attribute (at least partially) to website use. Those descriptions become a guide for using website analytic data to map particular patterns of use and to identify web use logs that can provide insight into how users may navigate the website.

Cool Trick: We use pop-up surveys to identify users that we might not otherwise reach. The pop-up asks for an email that we can use to follow-up with them for future surveys and interviews.

Want to learn more about Randahl and Ellen’s work? Join over 2500 colleagues at the AEA Annual Conference this November in San Antonio and check out their session in the conference program.

web analytics

My name is Jane Davidson and I run an evaluation consulting business called Real Evaluation Ltd. In my work, I advise and support organizations on strategic evaluation; provide evaluation capacity building and professional development; develop tools and templates to help organizations conduct, interpret, and use evaluations themselves; and conduct independent and collaborative evaluations and meta-evaluations.

Over several years’ working with clients and reviewing (at clients’ request) disappointing evaluation reports, I have noticed several critically important elements that make or break evaluation work but are often missing from evaluators’ methodological toolkits.

Hot tip: Clients find it incredibly frustrating to wade through an evaluation report full of evidence and still be none the wiser at the end whether the documented outcomes (let alone the entire program/policy/etc) are any good or not. A key part of an evaluator’s work is to say clearly and explicitly how practically, educationally, socially, or economically (not just statistically) significant outcomes are (severally, and as a set). This is what makes evaluation ‘e-VALU-ation’!

Hot tip: A useful tool for generating real evaluative conclusions is an evaluative rubric. This is a table describing what different levels of performance, value, or effectiveness ‘look like’ in terms of the mix of evidence on each criterion. Grading rubrics have been used for many years in student assessment. Evaluative rubrics make transparent how quality and value are defined and applied. I sometimes refer to rubrics as the antidote to both ‘Rorschach inkblot’ (“You work it out”) and ‘divine judgment’ (“I looked upon it and saw that it was good”)-type evaluations.

Hot tip: Collaborative development of rubrics is a great way to get stakeholders thinking about how ‘quality’ and ‘value’ should be defined for the work they do. It helps build the evaluative thinking needed to generate, understand, accept, and use evaluation findings.

Rad resources:

• Evaluation Methodology Basics: The nuts and bolts of sound evaluation by E. Jane Davidson (2005)
• Improving evaluation questions and answers: Getting actionable answers for real-world decision makers (AEA e-Library’s most viewed and downloaded item)
• Example rubrics in Nunns, Roorda, et al’s (2010) Evaluation of the Recognised Seasonal Employer Policy
• Example rubric (referred to as a ‘global assessment scale’) developed for the evaluation of the Corangamite Salinity Program (case study #10 in Jessica Dart et al’s 1998 Review of Evaluation in Agricultural Extension, pp. 62-63 – a publication from the Rural Industries Research and Development Corporation)
• AEA conference professional development workshop about how to use rubrics (and other evaluation nuts and bolts) to do actionable evaluations (November 10, 2010)
• Strategic evaluation of the workplace assessment program, a relevant recent chapter from Jane in the Handbook of Workplace Assessment (2010)

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org. Want to learn more from Jane? She’ll be presenting as part of the Evaluation 2010 Conference Program, November 10-13 in San Antonio.

Rubric, Toolkit

My name is Randahl Kirkendall. I am a public health manager turned evaluator. Platometrics is the name of my consulting business, which for the past three years has been focused on program research, planning, and evaluation. I am also a part-time evaluator for the Science Education Resource Center at Carleton College, which provides faculty professional development programs using a combination of workshops and web-based resources.

Four years ago while overseeing the development of two websites I learned how to use Google Analytics to track and measure website use. My first contract to evaluate website content was two years ago. Since then, I have learned much about evaluating program websites, but still consider myself to be on a steep learning curve in this area. Here is a little bit of what I have learned.

Lesson Learned: Using multiple and mixed evaluation methods that include both quantitative and qualitative metrics is the best way to fully understand the processes by which a website is being used as well as the outcomes that result. Web analytics can reveal much about how users navigate a website, which is something that users have difficulty recalling. Surveys and interviews can measure their motivations behind their website use, the impacts and outcomes of using a website, and descriptive information about the users themselves. Combining the two helps to provide a more complete picture that may also include the interplay between the website and other aspects of a program, such as a workshop or printed material.

Rad Resource: Occam’s Razor by Avinash Kaushik (www.kaushik.net/avinash). This website is built around a blog by an expert in web analytics who presents information in an easy to understand and good humored way. You might also want to check out his book, Web Analytics 2.0.

Hot Tip: I am currently developing a short Guide to Evaluating Program Websites, which I will post on www.platometrics.com later this month. If you would be interested in reviewing a draft or would like to be notified when it is posted, send me a note at rk@platometrics.com.

This is a relatively new and rapidly evolving area of evaluation, so if you know of any other good resources or ideas, please share them.

This contribution is from the aea365 Tip-a-Day alerts, by and for evaluators, from the American Evaluation Association. If you’d like to learn more from Randahl, consider attending his session at the AEA Annual Conference this November in San Antonio. Search the conference program to find Randahl’s session or any of over 600 to be presented.

Evaluation, mixed methods, web analytics

My name is Nina Potter and I am currently the Director of Assessment for the College of Education at San Diego State University. I’d like to share a little about a tool we are using for data visualization.

One of my responsibilities is to work with program directors and department chairs to evaluate academic programs across the college’s eight departments and 30+ programs. Our programs vary greatly in size and each has its own goals and student learning outcomes. Plus, we have some common goals across the college. We wanted to have a common tool that would allow us to share data across the college, but it had to be very flexible in terms of the kinds of data that it could handle as well as the kinds of reports that it could generate. After a lot of exploring, we chose Tableau.

Rad Resource: Before coming to SDSU, I had never heard of Tableau, in fact I had not heard the term “data visualization tool.” First I will tell you what it is NOT. Tableau is not a tool for data entry. You use Tableau to access data from other data sources such as spreadsheets or databases. This was important because our programs use many different tools to collect data, from electronic portfolio systems to paper and pencil tracking (we do require them to at least put the data in a spreadsheet). And, Tableau does not do advanced statistics; although it does do simple regression and t-tests. For statistical tests, we still use other statistic packages.

So what does Tableau do? Tableau allows you to link into multiple data sources, and quickly and easily create interactive graphs and charts that are updated in real time as your data sources are updated. It has a variety of choices for visualizations such as tables, line graphs, bar charts, pie charts and geographical maps. With just a few clicks you can easily change the type of chart, add colors, add filters and drill down to data that fits certain criteria. The charts are interactive so that anyone viewing the charts can apply filters and view the data they want to focus on.

For example, we have some assessments that are given across multiple programs. We can create a chart that looks at student progress over time and add filters such as program, gender, ethnicity, and age. A person who is evaluating the program as a whole can compare the results from program X to program Y to see if there is equity across multiple demographic groups. Additionally, a person who is working with individual students can download a list of students who have failed more than one assessment in a given program.

Want to hear more about Tableau from Nina? Join her on April 29 for “Data in, Brilliance Out with Tableau” as part of AEA’s Coffee Break Demonstration Series. More information and registration may be found at http://comm.eval.org/EVAL/coffee_break_webinars/Home/Default.aspx. Free for AEA members!

data visualization, software, technology