My name is Kristy Jang, and I am a Master’s student at the University of British Columbia, Canada. I am interested in evaluating educational programs in developing countries and professional development programs in higher education. Since last fall, I have had the pleasure to help evaluate a graduate-level professional development program which is based on a group-mentoring model. As part of our evaluation, we are looking at the development of social networks among stakeholders (e.g., trainees, mentors) and trainee achievements. In collecting data on these topics, we have encountered some helpful resources:

Rad Resource: First, Lime Survey is a free online survey development tool: www.limesurvey.org. You can collect data via e-mails with a link to the online survey, and the responses are automatically compiled within the server. You can export the data in different file formats (e.g., Excel, SPSS) and obtain data summary statistics and graphs. With regards to collecting social networks data, the most useful feature was “array_filter” function, which allowed us to ask questions with response choices that were filtered out from a previous question. For example, we first asked, “Which of the following people do you interact with?” Then, the respondent’s choices became the possible answer choices of the subsequent questions such as, “Whom do you ask for help when you have a challenging problem in your research?” and “Who do you talk to when you have a new innovative idea?” The social networks data were analyzed through UCINET software, which allowed us to visualize interactions among stakeholders as sociograms in three-dimensional space.

Rad Resource: Next, Google Docs is a helpful resource for simpler survey questions: www.gmail.com. Although it does not have complex features such as the “array_filter, it is more user-friendly with 71 design choices. Thus, it is more suitable for beginners and helpful for increasing response rates. Moreover, Google Docs Forms helps enhance communications among the evaluators as documents can be stored within the server and jointly edited by anyone who has access to the g-mail account.

Rad Resource: Adobe Acrobat 9 Pro software helps you collect information that could be better presented in a table rather than a survey format (e.g., a list of workshops and conferences respondents attended, including date, title, place, and the respondents’ roles). The software works similarly as the other online survey development tools. It sends out an e-mail with a link to the document, and when responses are submitted, it automatically compiles them in your computer as a separate file. Make sure that the PDF file you created is set to be writable, allowing respondents to save data in the document using Acrobat Reader – from the “Advanced” on the menu bar click on “Extend features in Acrobat Reader.”

Want to learn more about Kristy’s work? Join us at the American Evaluation Association’s Annual Conference, Evaluation 2010, in San Antonio this November and check out the poster exhibition on Wednesday evening.

google, software, survey

My name is Virginia Dick and I am currently public service evaluation faculty at the Carl Vinson Institute of Government at the University of Georgia. Most of my work focuses on assisting state and local government agencies, and other university divisions, with evaluation of programs, policies and systems.

As part of my role I often find myself working with a wide range of individuals with different backgrounds, perspectives, purposes, and information assessment styles. It has been important to find ways to help different groups examine and understand relevant evaluation data using a wide range of mechanisms.

Most recently, I have begun working with our state child welfare agency to use GIS (Geographic Information Systems) methods to examine child welfare client characteristics and outcomes spatially through mapping. Often key stakeholders (community members, agency leadership, and social work students) have expressed new and interesting perspectives and interpretations of the data when it is portrayed via mapping rather than in traditional charts and tables.

Rad Resource: ESRI (http://www.esri.com/) often provides free training and educational opportunities to work with their mapping software and may be available through some universities.

There are many open source software options out there, some of which I am currently working with at the University of Georgia Information Technology Outreach Service to explore with my current project. A list of open source options is available at: http://gislounge.com/open-source-gis-applications/

Hot Tip: When working with a group reviewing the data and relationships between variables, start with a few layers and options on the map and slowly build and add additional components as the individuals start to become more comfortable talking about the relationships between the different variables.

Hot Tip: By looking at census tracts as units it allows groups to discuss the relationship between variables without having to dig down to the individual street address level which can become much more complicated when compiling the maps. Often analysis at the census tract level can be most beneficial to communities and government agencies rather than the individual street address level.

Hot Tip: Let the stakeholders generate the ideas and discussion among themselves to get the richest information about the perceived relationship between variables. This is particularly useful when looking at small units such as counties or smaller (with the mapping done at the census tract or block level).

Want to learn more about Virginia’s work using GIS? Come to the poster exhibition on Wednesday evening in San Antonio this November for AEA’s Annual Conference.

census, child welfare, Geographic information systems, GIS, software

My name is Randahl Kirkendall. I work part-time as an Evaluator with Ellen Iverson, Director of Evaluation, for the Science Education Resource Center (SERC) at Carleton College, which works to improve education through projects that support educators.  Our work is funded primarily through NSF grants.  SERC has expertise in geoscience education, workshop leadership, website development and program and website evaluation.

A primary aim of SERC is to help faculty adopt evidence-based teaching behaviors that will enhance student learning. In evaluating the websites at SERC, our interest is in the role of website use in faculty professional development. We use a variety of web analytic tools such as Google Analytics, server-based website statistics, and web page visit logs in combination with data from surveys, interviews, focus groups, and observations to get as complete a picture as possible for how faculty use  websites and the impact that their use has on teaching behavior.

Lesson Learned: One of the things we have learned from user interviews is that people generally have poor recall of how they found a website and used it. While they can explain why they go to a website (motivation), they have difficulty recalling at what section of the website they started, what pages they viewed, and the search strategy they used. Website use analytics and web server logs of individual visits provide a richer picture of user behavior and interests via records of the actual pages that they visited.

Lesson Learned: The SERC websites often don’t work in isolation. Our survey of 2,000+ faculty found that a significant number of users were using the websites to compliment other professional development activities such as attending workshops, exchanging ideas with colleagues, or reviewing literature. Thus, it has been prudent that we collect data on these other possible influences on their teaching behavior.

Cool Trick: We sequence or build evaluations incrementally, partially basing data collection and/or analyses on findings from other data collection methods. For example, we use the findings from user interviews to describe predominant motivations for using a website and any changes in behavior (such as teaching practice changes) that users attribute (at least partially) to website use. Those descriptions become a guide for using website analytic data to map particular patterns of use and to identify web use logs that can provide insight into how users may navigate the website.

Cool Trick: We use pop-up surveys to identify users that we might not otherwise reach. The pop-up asks for an email that we can use to follow-up with them for future surveys and interviews.

Want to learn more about Randahl and Ellen’s work? Join over 2500 colleagues at the AEA Annual Conference this November in San Antonio and check out their session in the conference program.

web analytics

My name is Randahl Kirkendall. I am a public health manager turned evaluator. Platometrics is the name of my consulting business, which for the past three years has been focused on program research, planning, and evaluation. I am also a part-time evaluator for the Science Education Resource Center at Carleton College, which provides faculty professional development programs using a combination of workshops and web-based resources.

Four years ago while overseeing the development of two websites I learned how to use Google Analytics to track and measure website use. My first contract to evaluate website content was two years ago. Since then, I have learned much about evaluating program websites, but still consider myself to be on a steep learning curve in this area. Here is a little bit of what I have learned.

Lesson Learned: Using multiple and mixed evaluation methods that include both quantitative and qualitative metrics is the best way to fully understand the processes by which a website is being used as well as the outcomes that result. Web analytics can reveal much about how users navigate a website, which is something that users have difficulty recalling. Surveys and interviews can measure their motivations behind their website use, the impacts and outcomes of using a website, and descriptive information about the users themselves. Combining the two helps to provide a more complete picture that may also include the interplay between the website and other aspects of a program, such as a workshop or printed material.

Rad Resource: Occam’s Razor by Avinash Kaushik (www.kaushik.net/avinash). This website is built around a blog by an expert in web analytics who presents information in an easy to understand and good humored way. You might also want to check out his book, Web Analytics 2.0.

Hot Tip: I am currently developing a short Guide to Evaluating Program Websites, which I will post on www.platometrics.com later this month. If you would be interested in reviewing a draft or would like to be notified when it is posted, send me a note at rk@platometrics.com.

This is a relatively new and rapidly evolving area of evaluation, so if you know of any other good resources or ideas, please share them.

This contribution is from the aea365 Tip-a-Day alerts, by and for evaluators, from the American Evaluation Association. If you’d like to learn more from Randahl, consider attending his session at the AEA Annual Conference this November in San Antonio. Search the conference program to find Randahl’s session or any of over 600 to be presented.

Evaluation, mixed methods, web analytics

My name is LaMarcus Bolton, and I am the American Evaluation Association’s Technology Director, as well as a doctoral student in Industrial-Organizational Psychology. AEA’s Executive Director, Susan Kistler, is on vacation and I’m filling in with this Saturday’s staff post. Today, I wanted to discuss some of the great strides technology has made in terms of helping to improve communication, in particular focusing on quick, easy, and low-cost/no-cost tools that can help evaluators to keep their costs in check and maintain open communication with stakeholders and clients.

Rad Resource: Despite how far we have come in terms of communication mediums, the good ‘ol telephone is still often the medium of choice. Skype, allows one to make free calls to other Skype users, and has very competitive rates for both national and international calls for non-Skype users. Skype even allows one to create conference calls, although with a limited number of participants. However, if you do not want to rely on voice over Internet protocol (VOIP) technologies like Skype (which, by the way, typically are not as clear as traditional landlines), services such as FreeConferenceCall.com in the US may be worthwhile. I have used FreeConferenceCall.com on a few occasions and have always had pleasurable experiences.

Rad Resource: Although the wide availability of scanners and PDFs have made fax machines less popular, there is sometimes still the need to send or receive a fax. Services like eFax give users a free (though non-local) number that can accept faxes and send them directly to your email address. Unfortunately, to send faxes requires one to upgrade to their eFax Plus service. To address this, services like Fax Zero allows one to upload Microsoft Word and PDF files to send to any fax number in the country. Though these options are reliable, and environmentally friendly, I would caution against using them to send or receive sensitive information.

Rad Resource: If, like myself, you try to stay on the cutting-edge of technology, there have been great strides here as well. In addition to phone calls, Skype also allows for free video calls. This allows one to talk face-to-face with friends, family, and colleagues, by adding a small hint of personalization. Though, for quick and efficient communication, technologies like instant messaging (IM) are king. Though, the problem with instant messaging is that everyone seems to have a different preferred client. For example, popular here within the US are AOL Instant Messenger (AIM) and Google Chat (Gchat), but internationally, Windows Messenger is often preferred. To address this, software like Trillian and Digsby for Windows, and Adium for Mac, allow one to combine all IM services into a single and easy-to-use interface.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

communication, skype

I am Dana DeHart, Research Associate Professor at The Center for Child & Family Studies, a multidisciplinary evaluation and training unit within the University of South Carolina’s College of Social Work. I conduct grant-funded research dealing with violence and victimization. Some of my recent projects have included studies on incarcerated women, prior victimization, trauma histories of delinquent girls, and needs assessments for victim service initiatives, elder mistreatment prevention, and higher education in child welfare.

Hot Tips: I’m visually oriented, so when I’m planning or reporting on projects, I use mind-mapping software to help organize my thoughts. Wikipedia describes a wide variety of free and pay software applications for mind-mapping (Compendium, FreeMind, MindMapper, etc.). Typically, these allow the user to create tree-like networks of nodes with linked text, like the one below.

You can type text directly into a textbox corresponding to each node (see Effects on Well-Being, below), or you can cut and paste text from other programs such as your qualitative analytic software.

You can add or delete nodes, shuffle nodes around from branch to branch, and export the entire map to a graphic image file or to a word processing file that presents textual content as a report (see below).

Rad Resource: The mind-mapping software that I use is Mindjet MindManager. A free trial version is available at http://www.mindjet.com/index0.html

Mind Mapping, project management, software

I’m Susan Kistler, AEA’s Executive Director, and the author of each Saturday’s aea365 Tip. One challenge we have in the AEA office, that is echoed across those working with groups in almost any context,  is scheduling meetings (or scheduling focus groups or scheduling planning sessions, etc). Challenges include: (1) when we did this by hand with a spreadsheet, even the most diligent made mistakes; (2) it was time consuming and involved a lot of hand-entry, and correction and checking on time zones; and (3) potential attendees varied considerably in terms of the times that they would give (some giving only their convenient times, other giving all possible times including those that would require some wiggling around to make work).

We went in search of a software solution and found a number of time scheduling products on the market. After trying out three, we settled on one that has been a godsend.

Rad Resource: When is Good is a web-based tool that allows you to identify possible times for a meeting across multiple dates, and then send a link to your attendees so that they may choose which times are best. Attendees can click through and identify their available times in their own time zones. So far, these functionalities are common across multiple products and When Is Good does them very nicely in their free version with no signup fees and no registration needed. If you register, you can then create a dashboard where you can manage multiple scheduling events at once, all still free. Thus, the free version solved challenges #1 and #2 from above and was incredibly useful. However, what really made it stand out, for our office, was that when you pay for the premium version, at 5 pounds a year (about US$6.50), your attendees can choose those times that are best, ok, or possible but not ideal. This addressed challenge #3 and has greatly increased the range of times that our colleagues indicate availability, increasing the likelihood that we can identify a common time even across a large group.

Rad Resource: XE.com is an easy to use currency converter site. When Is Good is developed by a family-run software company in the UK (we’ve corresponded with the very nice developer to ask a question or two). Their payment options are all in pounds. Wanting to be sure that we were clear about the extent of our investment, XE allowed for a quick check-in and confirmation that we could, indeed, afford $6.50 for a year of When Is Good.

The above opinions are my own and do not necessarily reflect those of AEA.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

time management

My name is Susan Kistler, AEA’s Executive Director, and I contribute each Saturday’s aea365 post. Earlier this week, I had the pleasure of speaking about the democratization of data at the AEA/CDC Summer Evaluation Institute.  While a candidate, US President Barak Obama committed to putting government data online in universally accessible formats (see here). At a 2009 TED Talk, Tim Berners-Lee – the founder of the Internet – urged the audience to share “Raw Data Now” (see here). And just this year, statistician and activist Hans Rosling noted that times have changed and we need to get data to the public (see here). Taken together, the message is clear – we’re in a new age of data sharing. But what does this mean for evaluation?

Lessons Learned for Evaluators:

• There are competing forces at work, with competing expectations and needs. As an example, the call for open data sharing can be in conflict with the Health Information Privacy Act (HIPPA) which protects individual privacy.
• We may need to bring new people on our evaluation teams in order to create mechanisms for the public to access and use data in ways that promote utility and understanding. One strong example comes from the Kids Count Data Center, an initiative of the Annie E Casey Foundation that makes measures of child well being accessible to the public.
• The US Government is making strides towards realizing the Obama vision for online data sharing via such initiatives as data.gov – but weaker progress in terms of making data available in universally accessible formats or in ways that are understandable not only to researchers but also to the general public.
• Making data publicly available can result in intriguing use such as the recent competition from The Sunlight Foundation that provided a $5,000 prize from the US Department of Health and Human Services for “innovative applications that improve the public’s understanding of community health performance” based on a DHHS dataset. Type your US postal code into the winner “County Sin Rankings” for a potentially surprising look at your locale.
• Interactive interfaces and tools that allow the general public to analyze data and create a near-infinite number of data visualizations, can produce results with subtleties that may not be apparent to the general consumer. Take a look at these two maps of infant mortality rates from the New York Times Data Visualization Lab, comparing rates from 1960 and 2004 in selected countries. You’ll need to look carefully at the scales to see the considerable difference.

Rad Resource: This downloadable session handout provides links to 20 tools for making data publicly accessible – from examples of data in use to data repositories to free tools for collecting, analyzing, and visualizing data.

The above opinions are my own and do not necessarily reflect those of the American Evaluation Association.

Data, Database management

Hi, my name is Bi Deng. I am a doctoral student at Claremont Graduate University (CGU) studying Organizational Behavior with a co-concentration in Evaluation. Back in 2009, I attended a CGU workshop by our GIS specialist Warren Roberts and was blown away by the utility of GIS in social science research, particularly for evaluation projects. I’ll be talking about how GIS can help us answer a wide range of place-, space-, and time-oriented questions (Meeks, 2008).

What is GIS? A geographic information system (GIS) captures, stores, analyzes, manages, shares, and presents data that is linked to location, such as people to an address or crime to a demographic area. GIS allows us to view, interpret, and visualize data in ways that reveal relationships, patterns, and trends in the form of maps, globes, reports, and charts (ESRI.com).

Why use GIS? If you were to look at a tabular set of data, it is not easy to interpret or see a pattern in a large database. GIS allows you to visualize large amounts of complex, spatial data across layers of various maps. Most problems and issues (e.g., cultural, environmental, economic, political, social) we face today exist in a geographic context (ESRI.com). An estimated 80% of all data held by business and government organizations have a spatial component (Franklin, 1992) – which means most data in can be analyzed “spatially.”

Some of the basic research questions GIS can help us answer include (W. Roberts, personal communication, January 23, 2009):

• Location: What is at…?
• Trends: What has changed…?
• Condition: Where is the location that meets these criteria?
• Pattern: What is the spatial pattern or distribution…?
• Modeling: What if…?

For example, when doing spatial analysis, you can run queries to:

• Identify specific features (e.g., what is the income level of a particular region that your client is serving)
• Identify features based on conditions (e.g., out of a group of regional sites, you can identify all the high-performing or low-performing sites of your client)
• Interactions (e.g., investigate whether poverty levels of a region affects the participation rates of your client organization)

Rad Resource: If you would like to learn more about GIS, the ESRI website is a great resource. ESRI’s ArcGIS community posts featured maps and applications of GIS. On ArcGIS.com, you can take a quick tutorial and register for a free account to create and share an interactive map: http://www.arcgis.com/home/gallery.html

Rad Resource: Want to explore what kinds of free data there is for researchers? Check out these sites:

• http://gisinfo.libraries.claremont.edu/dataresources.asp

• http://data.geocomm.com/

Rad Resources:

• Franklin, C. (1992). An introduction to geographic information systems: Linking maps to databases. Database, 15(2), 17-22.

• Meeks, W. L. (2008). The utility of geospatial data and information used in Geographic Information Systems (GIS): An exploratory study into the factors that contribute to geospatial information utility. Unpublished doctoral dissertation, George Washington University, Washington, DC.

Cool Trick: Want to see a neat example of GIS in action? Check out Recovery.com – this site uses GIS to track and display data related to how funds of the Recovery Act is being awarded and spent. You can check to see how $275 billion of federal funds made available for federal contracts, grants, and loans are being used in your own community. http://www.recovery.gov/Pages/home.aspx

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

GIS

My name is Susan Kistler and I am AEA’s Executive Director. I contribute each Saturday’s post to the aea365 blog. This week, Lois Ritter and Tessa Robinette gave a great (free!) webinar for AEA comparing Surveymonkey and Zoomerang, helping to compare and contrast the two for potential new users. Building on their presentation, and from my own experiences having worked with both of these programs as well as two others over the past two years, I wanted to share this week a tip and lesson learned in using online survey software.

Hot Tip: Take Advantage of the Free Trial – The majority of survey platforms offer a free trial, usually allowing for only a few questions and respondents. Create and try out a sample survey from beginning to end, including sending invitations, collecting data, and completing the analysis, usually by exporting the data into other software for further analysis. Walk through the entire process with data parallel to that which you anticipate for your actual survey.

Lessons learned – Permissions: Does your survey platform help you to comply with IRB expectations? With the CAN-SPAM act? While using the built-in invitation functions of many online survey platforms can help you with sending, tracking, and compliance, it can also distort your sample and limit your access to potentially viable respondents because opt-out treatment varies from platform to platform. Using the two platforms discussed in this week’s webinar as examples, when survey recipients opt-out of a SurveyMonkey survey invitation, by clicking the opt-out button on the bottom, they are directed to this address http://www.surveymonkey.com/optout.aspx and they opt-out of ALL surveys from SurveyMonkey, not just a particular survey or those from a particular sender. Alternatively, when users opt out of a survey from Zoomerang, the default is that they opt out of surveys only from that particular sender (see http://zoomerang.custhelp.com/app/answers/detail/a_id/308). Notably though, it is much easier to opt back in to SurveyMonkey than Zoomerang. Lesson learned? Research how your platform treats opt-outs and determine how this is likely to impact your respondent pool.

Hot Tip: If you are an AEA member, review the Ritter and Robinette Webinar Recording in the AEA webinars archive at http://ow.ly/1XqAh to gain a better understanding of other considerations when choosing a survey platform.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org. The above comments reflect my own opinion and not necessarily that of the American Evaluation Association.

opt-out, permissions, surveymonkey, webinar, zommerang