AEA365 | A Tip-a-Day by and for Evaluators

TAG | software

Hi! My name is Laura Sefton, Research Coordinator at the University of Massachusetts Medical School’s Center for Health Policy and Research. My colleagues and I use Atlas.ti qualitative data analysis software to facilitate our program evaluation work. Atlas is one of several software packages available to evaluators. The Center chose to purchase Atlas (1) to standardize our work across the organization thus facilitating collaboration and (2) because many were already Atlas users, thus minimizing the number who’d learn new software. Based on our experiences, I’ve put together some tips and resources which may be helpful to new users or those considering purchasing qualitative analysis software.

Hot Tip: Three great Atlas.ti functionalities our project teams have used:

  • Organize documents into families so you can use the ‘filter’ function to see just the family of documents you want. This robust software allows users to work with many types of documents, including text, images, video, and audio files. Filtering allows you to put disparate project documents into the same HU (Hermeneutic Unit-the file that contains the linked project documents) and choose to look at one set or type of documents.
  • Create memos to track your work or thoughts as you move through your data analysis. All users can create and iteratively edit memos to share information with the team.
  • Add comments to your coded text. This is another method of tracking or sharing your thoughts with others. I’ve used this to tell my colleagues why I coded the text in a particular way. They can edit the comment to let me know if they agreed with my logic or have other thoughts.

Lesson Learned: Documents appear in the ‘Primary Document’ pane in the order in which they were added to the HU but they can be reorganized using the Renumber and Change Positions functions.

Lesson Learned: Atlas allows users to make edits or other text changes to a document after it’s been assigned to the HU. Use the Document Access function under the Edit menu to make the changes.

Rad Resource: Browse the Atlas.ti website to explore training options, download user manuals, sign up for the mailing list, or join the users’ forum. Training options range from free webinars to paid in-person consultants. Atlas offers single, student and a variety of multi-user licenses for commercial, educational, or government organizations.

Rad Resources: Advance preparation can make the coding and analysis work easier and faster. These previous AEA365 blog posts can help you:

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hi, I’m Craig Wiles, Senior Consultant at Public Sector Consultants in Lansing, Michigan. I provide research and evaluation services for clients in health and human services, education and the environment. I am sharing a tip on how to use Tableau as part of a data exploration process with a group of stakeholders.

To begin, I did the heavy statistical lifting outside of Tableau, so this would not lapse into a data-mining exercise. In this case, I worked with a state-level stakeholder group to identify data sources, research priorities, and statistically significant correlations in the data. Once we had our short list of correlated variables to explore in more detail, we convened a series of two hour, interactive data exploration sessions. At these sessions, we used Tableau to visually display the data (in this case, educational data), identify high and low performing school districts, and look for other obvious patterns or outliers in the data.

We tended to use stacked bar charts and scatter plots to help with this visual part of the data exploration. One tool in Tableau that was especially helpful in this context was the filter bar. Using the filter tool, we were able to adjust the range of scores we were looking for in our combination of variables according to tolerances set by the stakeholders. For example, we looked for school districts that had a high graduation rate, low dropout rate, and a higher ratio of students with disabilities in general education classrooms.

I recommend using Tableau for data exploration because it is:

  • Visual,
  • Interactive, and
  • Builds capacity and ownership.

I could have presented this data in charts and graphs and led a typical ‘sit-and-get’ meeting and landed at the same place (conceptually) at the end of the day. This kind of visual and interactive process, however, really helped to engage my stakeholders, especially those that are usually averse to numbers and data. Ultimately, this was as much about the process as it was about the data.

After our interactive sessions, this group began a series of local focus group conversations with voluntary school districts to further explore the relationships we identified. This qualitative data has provided depth of detail and rich context to the quantitative relationships we explored together.

Using the filter tool:

Tableau filter tool

Using a scatter plot:

Tableau scatter plot

Click to increase size

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

Greetings, I am June Gothberg, Lead Curator for aea365 and Research Associate at Western Michigan University. As Lead Curator, I am always looking for ways to expand the knowledge of evaluators through hot tips, cool tricks, lessons learned, and rad resources. While working on my dissertation, I made a great find and thought I would share it with you. I was looking for a way to measure my variables measuring conversations between participants. In my search of the literature, I ran across the Linguistic Inquiry and Word Count (LIWC) software and discovered it’s multitude of uses.

Language Inquiry and Word Count Software

LIWC is a computerized text analysis program with Mac and Window versions. It calculates the degree to which people use different categories of words across texts, including emails, speeches, poems, or transcribed daily speech. A few of the most interesting include positive or negative emotions, self-references, causal words, as well as 70 other language dimensions. A new area in which LIWC is being used is social network analysis.

Lessons Learned:

  • Don’t be afraid to go outside your field. For example, the roots of modern text analysis are found in the field of psychology.
  • In general, LIWC categorizes words hierarchically. For example, insight is a subgroup of cognitive processes and anger is a subgroup of negative emotions. So, you must decide what level to measure.
  • LIWC offers a nice triangulation for analyzing data. It helped validate the rater/coder findings of my study in an unbiased manner.
  • Except for raw word count and words per sentence, all variables reflect the percentage of total words.

Hot Tips:

  • LIWC offers a truncated free online version. This is a good way to try-before-you-buy. You must supply the gender and age for the participant from which the text was derived.

LIWC free version

  • LIWC allows customized dictionaries of words and phrases. We are currently working on an evidence-based dictionary to identify words in speech as markers for resiliency and self-determination.
  • Read the manual! The manual explains how to deal with abbreviations, punctuation, numerals, contractions, time stamps, slang, nonfluencies, and filler words.
  • Use a transcriber who understands the manual. If transcriptionists follow the LIWC guidelines much time and effort is saved.
  • Use the option for batch processing.
  • Combine variables. If you have a certain variable of interest, you may move LIWC output into your statistical analysis software and combine variables. One of our variables of interest was participant feelings of a positive employment outlook. We combined positive emotion, future tense, and employment (posemo+future+work). We were then able to compare those who participated in a skills training session and those who did not.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

May/11

31

LaMarcus Bolton on Online Security

My name is LaMarcus Bolton, and I am the American Evaluation Association’s Technology Director, as well as a doctoral student in Industrial-Organization Psychology. The majority of us use computers and the Internet on a regular basis for our work and personal lives. However, with the dependence on such technologies comes great risks to our data. Today, I wanted to tell you about some excellent resources for maintaining your online security.

Rad Resource: PortableApps is a wonderful website that houses a wide array of open source software and freeware portable apps. Portable apps are an excellent way to increase your cyber-security. You simply install the application onto a flash drive. When you need to use the application, you run it directly off the flash drive instead of installing it onto your computer. This is particularly an excellent option when you are using public computers. By using portable apps, all data is saved on the flash drive; no information is left behind on the computer itself. There are so many portable apps available for all of your needs including: web browsing, email, instant messaging, PDF-reading, word processing, spreadsheets, presentations, drawing, databases, calendars, tasks, etc.

Rad Resource: If you are on a computer that does not have anti-virus software, it can be difficult to tell if a file or link is infected. You also may not want to (or even be authorized) to download software on the machine as well. VirusTotal is a free online service that analyzes files and URLs to detect viruses, worms, trojans, and other kinds of malware. You simply upload the file or enter the URL and start scanning.

Rad Resource: Today, our online identities are spread across so many sites that remembering all of your passwords (especially when some services have different requirements) can be difficult. Plus, there are an increasing number of websites created to “phish” for your passwords that makes password management even more crucial. 1Password, while premium software, provides 1-click filling and highly-secure password management that remedies these two issues. It is compatible with Windows, Mac, and multiple smartphone operating systems.

Rad Resource: Have you ever wanted to send a private or confidential message to someone without the use of an insecure medium such as email? ThisMessageWillSelfDestruct is a website that allows users to write encrypted messages. The website will create a URL for accessing the message one time only. As soon as the message is viewed, it is deleted. Neither the sender or receiver is able to view the message again. Users also have the option of password-protecting the message for extra security, if they choose to do so.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

My name is Kristy Jang, and I am a Master’s student at the University of British Columbia, Canada. I am interested in evaluating educational programs in developing countries and professional development programs in higher education. Since last fall, I have had the pleasure to help evaluate a graduate-level professional development program which is based on a group-mentoring model. As part of our evaluation, we are looking at the development of social networks among stakeholders (e.g., trainees, mentors) and trainee achievements. In collecting data on these topics, we have encountered some helpful resources:

Rad Resource: First, Lime Survey is a free online survey development tool: www.limesurvey.org. You can collect data via e-mails with a link to the online survey, and the responses are automatically compiled within the server. You can export the data in different file formats (e.g., Excel, SPSS) and obtain data summary statistics and graphs. With regards to collecting social networks data, the most useful feature was “array_filter” function, which allowed us to ask questions with response choices that were filtered out from a previous question. For example, we first asked, “Which of the following people do you interact with?” Then, the respondent’s choices became the possible answer choices of the subsequent questions such as, “Whom do you ask for help when you have a challenging problem in your research?” and “Who do you talk to when you have a new innovative idea?” The social networks data were analyzed through UCINET software, which allowed us to visualize interactions among stakeholders as sociograms in three-dimensional space.

Rad Resource: Next, Google Docs is a helpful resource for simpler survey questions: www.gmail.com. Although it does not have complex features such as the “array_filter, it is more user-friendly with 71 design choices. Thus, it is more suitable for beginners and helpful for increasing response rates. Moreover, Google Docs Forms helps enhance communications among the evaluators as documents can be stored within the server and jointly edited by anyone who has access to the g-mail account.

Rad Resource: Adobe Acrobat 9 Pro software helps you collect information that could be better presented in a table rather than a survey format (e.g., a list of workshops and conferences respondents attended, including date, title, place, and the respondents’ roles). The software works similarly as the other online survey development tools. It sends out an e-mail with a link to the document, and when responses are submitted, it automatically compiles them in your computer as a separate file. Make sure that the PDF file you created is set to be writable, allowing respondents to save data in the document using Acrobat Reader – from the “Advanced” on the menu bar click on “Extend features in Acrobat Reader.”

Want to learn more about Kristy’s work? Join us at the American Evaluation Association’s Annual Conference, Evaluation 2010, in San Antonio this November and check out the poster exhibition on Wednesday evening.

· ·

My name is Kai Chi Yam, and I am a graduate student at Washington State University. I recently had an opportunity to evaluate a university-based mentoring program. Exploratory Factor Analysis (EFA) was used to provide score validity evidence through the examination of internal structure of study-specific measures. Such evidence can increase the utility of the instruments in evaluation, and ultimately increase the creditability and utility of evaluation results. While working with my colleagues, I found that some evaluators may be familiar with EFA, but most follow software default settings with little consideration of different procedures that can affect the results of EFA. Here are some general guidelines for appropriate use of EFA.

Hot Tip: Follow these steps when conducting an EFA:

1. Data screening: Check Chi Square (p > .05), Kaiser-Meyer-Olkin measure of sampling adequacy (MSA > .70), Bartlett’s test of sphericity, univariate and multivariate normality to determine if the data is appropriate for factor analytic procedures.

2. Extraction methods: Use principal component analysis if the rationale of EFA is purely data reduction. Use principal axis factoring if the rationale of EFA is to extract latent variables.

Note: Principal axis factoring is preferred because it takes measurement error into account.

3. Rotation methods: Use direct oblimin or promax rotation when factors are non-orthogonal. Use varimax rotation when factors are orthogonal.

Note: In most evaluations, especially when the measurements are psychological in nature, evaluators should assume factors to be non-orthogonal and proceed with direct oblimin or promax rotation.

4. Criteria for retaining factors: Decide the number of factors based on at least two criteria (e.g., scree plot, parallel analysis, conceptual meaningfulness of the factor) other than Kaiser’s rule of eigenvalue > 1.

5. Interpretation: Name each factor with theoretical (i.e., supporting theories) and statistical justifications (i.e., factor loadings). A general rule of thumb for acceptable factor loading is .40 or above.

Note: All interpretations must be theoretically justifiable! Don’t base your judgment solely on factor loadings.

Rad Resource: Making Sense of Factor Analysis: The Use of Factor Analysis for Instrument Development in Health Care Research. (ISBN: 0761919503) This introductory textbook is easy to follow and requires minimum knowledge in statistics and math.

Exploratory factor analysis can be a useful tool in evaluation when study-specific measures are employed. Please note that these are general guidelines, not definitive rules. Evaluators should consult with methodologists, textbooks, or journal articles before attempting EFA.

Want to learn more about Kat Chi’s work using EFA? Join us at the American Evaluation Association’s Annual Conference, Evaluation 2010, in San Antonio this November and check out the poster exhibition on Wednesday evening.

· ·

My name is Virginia Dick and I am currently public service evaluation faculty at the Carl Vinson Institute of Government at the University of Georgia. Most of my work focuses on assisting state and local government agencies, and other university divisions, with evaluation of programs, policies and systems.

As part of my role I often find myself working with a wide range of individuals with different backgrounds, perspectives, purposes, and information assessment styles. It has been important to find ways to help different groups examine and understand relevant evaluation data using a wide range of mechanisms.

Most recently, I have begun working with our state child welfare agency to use GIS (Geographic Information Systems) methods to examine child welfare client characteristics and outcomes spatially through mapping. Often key stakeholders (community members, agency leadership, and social work students) have expressed new and interesting perspectives and interpretations of the data when it is portrayed via mapping rather than in traditional charts and tables.

Rad Resource: ESRI (http://www.esri.com/) often provides free training and educational opportunities to work with their mapping software and may be available through some universities.

There are many open source software options out there, some of which I am currently working with at the University of Georgia Information Technology Outreach Service to explore with my current project. A list of open source options is available at: http://gislounge.com/open-source-gis-applications/

Hot Tip: When working with a group reviewing the data and relationships between variables, start with a few layers and options on the map and slowly build and add additional components as the individuals start to become more comfortable talking about the relationships between the different variables.

Hot Tip: By looking at census tracts as units it allows groups to discuss the relationship between variables without having to dig down to the individual street address level which can become much more complicated when compiling the maps. Often analysis at the census tract level can be most beneficial to communities and government agencies rather than the individual street address level.

Hot Tip: Let the stakeholders generate the ideas and discussion among themselves to get the richest information about the perceived relationship between variables. This is particularly useful when looking at small units such as counties or smaller (with the mapping done at the census tract or block level).

Want to learn more about Virginia’s work using GIS? Come to the poster exhibition on Wednesday evening in San Antonio this November for AEA’s Annual Conference.

· · · ·

I am Dana DeHart, Research Associate Professor at The Center for Child & Family Studies, a multidisciplinary evaluation and training unit within the University of South Carolina’s College of Social Work. I conduct grant-funded research dealing with violence and victimization. Some of my recent projects have included studies on incarcerated women, prior victimization, trauma histories of delinquent girls, and needs assessments for victim service initiatives, elder mistreatment prevention, and higher education in child welfare.

Hot Tips: I’m visually oriented, so when I’m planning or reporting on projects, I use mind-mapping software to help organize my thoughts. Wikipedia describes a wide variety of free and pay software applications for mind-mapping (Compendium, FreeMind, MindMapper, etc.). Typically, these allow the user to create tree-like networks of nodes with linked text, like the one below.

You can type text directly into a textbox corresponding to each node (see Effects on Well-Being, below), or you can cut and paste text from other programs such as your qualitative analytic software.

You can add or delete nodes, shuffle nodes around from branch to branch, and export the entire map to a graphic image file or to a word processing file that presents textual content as a report (see below).

Rad Resource: The mind-mapping software that I use is Mindjet MindManager. A free trial version is available at http://www.mindjet.com/index0.html

· ·

My name is Nina Potter and I am currently the Director of Assessment for the College of Education at San Diego State University. I’d like to share a little about a tool we are using for data visualization.

One of my responsibilities is to work with program directors and department chairs to evaluate academic programs across the college’s eight departments and 30+ programs. Our programs vary greatly in size and each has its own goals and student learning outcomes. Plus, we have some common goals across the college. We wanted to have a common tool that would allow us to share data across the college, but it had to be very flexible in terms of the kinds of data that it could handle as well as the kinds of reports that it could generate. After a lot of exploring, we chose Tableau.

Rad Resource: Before coming to SDSU, I had never heard of Tableau, in fact I had not heard the term “data visualization tool.” First I will tell you what it is NOT. Tableau is not a tool for data entry. You use Tableau to access data from other data sources such as spreadsheets or databases. This was important because our programs use many different tools to collect data, from electronic portfolio systems to paper and pencil tracking (we do require them to at least put the data in a spreadsheet). And, Tableau does not do advanced statistics; although it does do simple regression and t-tests. For statistical tests, we still use other statistic packages.

So what does Tableau do? Tableau allows you to link into multiple data sources, and quickly and easily create interactive graphs and charts that are updated in real time as your data sources are updated. It has a variety of choices for visualizations such as tables, line graphs, bar charts, pie charts and geographical maps. With just a few clicks you can easily change the type of chart, add colors, add filters and drill down to data that fits certain criteria. The charts are interactive so that anyone viewing the charts can apply filters and view the data they want to focus on.

For example, we have some assessments that are given across multiple programs. We can create a chart that looks at student progress over time and add filters such as program, gender, ethnicity, and age. A person who is evaluating the program as a whole can compare the results from program X to program Y to see if there is equity across multiple demographic groups. Additionally, a person who is working with individual students can download a list of students who have failed more than one assessment in a given program.

Want to hear more about Tableau from Nina? Join her on April 29 for “Data in, Brilliance Out with Tableau” as part of AEA’s Coffee Break Demonstration Series. More information and registration may be found at http://comm.eval.org/EVAL/coffee_break_webinars/Home/Default.aspx. Free for AEA members!

· ·

My name is Susan Kistler and I am the Executive Director for the American Evaluation Association. I contribute each Saturday’s post to the aea365 blog. This week I am writing from Atlanta at the Nonprofit Technology conference.

Do you work for or with nonprofit organizations? Have you experienced challenges due to financial constraints that make technology purchases for evaluation beyond the budget?

Hot Tip: Take a look at TechSoup, the “technology place for nonprofits.” TechSoup has resources, training, a peer-to-peer community, and a donated technology program – TechSoup Stock. Their donated tech program gives nonprofits access to products from a range of big name (and not so big name) companies. Examples include the full Microsoft Office Suite including Access and Excel; ArcGIS from ESRI for spatial analysis; and Crystal Reports from SAP for data visualization and reporting. And the cost? Each product has an administrative fee, but most are well below even discounted retail prices. As an example, the full Microsoft Office 2007 suite is $20. Organizations do need to go through a relatively painless qualification process, and the eligibility criteria vary from product to product, but the resource is definitely worth checking out.

The opinions expressed above are my own and do not necessarily represent those of my employer, the American Evaluation Association.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

· · · · · · ·

Archives

To top