AEA365 | A Tip-a-Day by and for Evaluators

TAG | Data

Our names are Lindsey Dunn and Lauren Fluegge and we are PhD. students at The University of North Carolina at Greensboro. Recently, we have been involved with a project in which the data come from many different sources and often include hundreds of thousands of cases. We used primarily three programs to organize and merge the data (Microsoft Excel 2007, Microsoft Access 2007, and PASW 18), and include here pros and cons of each program and information about the advantages of the statistical program R.

Excel 2007:

Pros

  • Sorts/filters data efficiently
  • Summarizes data with Microsoft PivotTable, which automatically sorts, counts, totals or averages the data

Cons

  • Analyses excluded from the formula repertoire and results formatting must be entered by hand
  • Has “behind the scenes” syntax out of the user’s control which makes it difficult to reproduce procedures on different data

Hot Tip: The text to column function of Excel is useful to reformat identifiers, such as names, to match between two data files you want to merge

Access 2007:

Pros

  • Relational nature of databases allows for flexible linking of data files (e.g. the user can match multiple data files by variables with different names; a smaller subset of variables of interest can be selected from the original data files to be in a newly merged data file)
  • Access cannot analyze data

Cons

  • Merging can be slow or freeze the program if done with large data files
  • Syntax is out of the user’s control

Hot tip: Choose “Export” to move any data files to Excel for formatting and simple calculations

PASW 18 (SPSS):

Pros

  • Can accomplish a plethora of analyses, decent graphics, and easy to read output
  • Efficiently identifies duplicates, sorts, and filters data
  • Can be controlled by user-written syntax

Cons

  • Inflexible merging procedures
  • Error messages are not always clear
  • Expensive

Hot Tip: With SPSS, can now open files without a .sav with the “Open” command and retain the field names (rather than using the “New Query” command)

Hottest Tip: Use R statistical program

  • R is more efficient than other programs because it allows you to do everything in one place!
  • In R, you have control over what and how you merge (e.g. can merge multiple data files at once by several variables)
  • You can manage and analyze data as effectively as the other programs in the same program
  • R is free and once you have written a merging protocol it can be reused with other data files!

Rad Resources:

Useful Rad resources are listed below:

http://stat.ethz.ch/R-manual/R-patched/library/base/html/merge.html

http://stat.ethz.ch/R-manual/R-devel/library/base/html/duplicated.html

http://www.r-project.org/

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

· · · ·

My name is Guili Zhang. I am an Assistant Professor of Research and Evaluation Methodology at East Carolina University. During the last ten years, I have evaluated the National Science Foundation’s SUCCEED program, and developed and analyzed the SUCCEED longitudinal database, which includes data from nine universities and spans 20 years. Our research team’s publications based on this database have received two Best Paper Awards from the American Society of Engineering Education and the Frontiers in Education. Today I’d like to share some information about longitudinal data management and analysis.

Lessons Learned: There are two very different organizations for longitudinal data—the “person-level” format and the “person-period” format. A person-level data set, also known as the multivariate format, has as many records as there are people in the sample. As additional waves of data are collected, the file gains new variables, not new cases. A person-period data set, also known as the univariate format, has multiple records for each person—one for each person-period combination. As additional waves of data are collected, the file gains new records, but not new variables.

Besides the derived variable approach to longitudinal data analysis, which involves the reduction of the repeated measurements into a summary variable, there are two classical approaches: the ANOVA and MANOVA approaches. The ANOVA and MANOVA approaches represent well-understood methodology, and the computer software is widely available. Unfortunately, both models have limited usage in longitudinal data analysis due to their restrictive and often unrealistic assumptions and the effect of missing data on the statistical properties of their estimates. Currently, there are several alternative approaches that overcome the limitations of the traditional approaches, variously known as: mixed-effect regression model, the covariance pattern model, generalized estimating equations model, individual growth model, multilevel model, hierarchical linear model, random regression model, survival analysis, event history analysis, failure time analysis, and hazard model.

Hot Tip #1 – The person-period format most naturally supports meaningful analysis of change over time.

Hot Tip #2 – Most statistical software packages can convert a longitudinal data set from one format to another. For example, in SAS, Singer (1998, 2001) provides simple code for the conversion; in STATA, the “reshape” command can be used.

Rad Resources:

Two introductory books that I have found useful are:

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

My name is Chris Lysy and I am a Research Analyst at Westat. I am the creator of EvalCentral and I also blog at Fresh Spectrum. Additionally, you can find me on Twitter: @clysy.

One promising new online trend is the addition of data visualization tools on major data source websites. So often data is released in either simplified reports or large raw data-sets; reports leave out a lot of the specifics (where the context lives) and while the raw data will eventually provide you with what you need, getting there can be tough. Visualization provides an opportunity to connect these extremes by offering a mechanism to find specific contextual data in a user-friendly visual format. The following are some examples.

Rad Resource: United States Census Bureau

The Census Bureau has accompanied its first release of data from the 2010 Census with a nice interactive widget.  The base view provides a color coded map of the United States along with a chart. You can choose to view one of three metrics; population change, population density, or apportionment.   To see state level data simply use your mouse to point to the individual state.  The widget also gives you a chance to see the same data for past census years.

Rad Resource: World Bank Data

If you’re a data person, you could probably spend hours just clicking around the different visualized indicators on the World Bank Data site. The dashboard format includes maps, embeddable graphs, and data tables equipped with sparklines (for more on Sparklines, see Larua Blasi’s January 11 post).

Rad Resource: United States Geological Survey (USGS)

While maybe not as sleek looking as the previous resources, the USGS provides real-time monitoring visualization of earthquakes and other hazards. If you ever feel a tremor check this site first – they will beat any news organization.

Hot Tip: If you ever decide to create your own data dashboard, check out Juice Analytics’ “Guide to Creating Dashboards People Love to Use.” It provides a nice little overview of the dos and don’ts of dashboard design.

The American Evaluation Association is celebrating Data Visualization and Reporting Week with our colleagues in the new DVR AEA Topical Interest Group. The contributions all this week to aea365 come from our DVR members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DVR resources. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

· · · · ·

My name is Stephanie Evergreen and I am so proud to be the founding Chair of the newest Topical Interest Group in AEA. What new TIG, you ask? It’s the Data Visualization and Reporting TIG. And we are so geeked to be here that we’ve taken over a whole week of aea365 to share with you some of our favorite ideas and resources around data visualization and reporting.

Hot Tip: What do we mean when we are talking about data visualization? In short, it is the graphic representation of data, a visual communication. Want to see what I mean visually? Check out the Periodic Table of Visualization Methods. Mouse over a cell to get an example of visualizations of all sorts.

Hot Tip: Check out the most current buzz in data visualization by searching on this hashtag in Twitter: #dataviz. You don’t even have to have a Twitter account to be able to search for the latest developments in data visualization. Here’s a sampling of the awesome resources I found when I searched #dataviz at the time of this writing:

Rad Resource: Evaluators mainly work in written communication. Why not use the available tools of graphic design to make our work ever easier to read? In this cool cheat sheet, you can quickly scan for legible typefaces in good combinations, one for the main text of your evaluation report, and one for the headings and titles.

Rad Resource: I just love what happens after I visit www.vizthink.com. This website has webinars, podcasts (hint: download and listen on the way to work), and tons of examples of great infographics and slideshows. When I browse their site, I invariably find inspiration and new ways to bring my evaluation work to the next level.

Hot Tip: If all this talk on data visualization and reporting is intriguing to you, consider joining the TIG. AEA members can select to join up to 5 TIGs. To join DVRTIG, login to www.eval.org. Then, under the Members Only tab, select Update My Profile. In your profile, you simply check the box next to our TIG’s name. That’s it! You’re a member!

We started DVRTIG because it is increasingly clear that evaluators will have to be more creative in our communications if we want our findings to be heard and used. But, like you, I don’t have time to surf the web all day for great resources. That’s why I’m so glad we have a network of like-minded evaluators who can support each other. This week of DVRTIG blogging on aea365 is such a great start.

The American Evaluation Association is celebrating Data Visualization and Reporting Week with our colleagues in the new DVR AEA Topical Interest Group. The contributions all this week to aea365 come from our DVR members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DVR resources. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

·

Hello, my name is Melissa Maras, and I am an assistant professor at the University of Missouri. Schools are complex, interdisciplinary contexts always in flux with new and evolving policies, programs, and practices, all resulting in a rich mess of data that should be used in school-based evaluation, but is often difficult to navigate. Below are some ideas and resources that may be helpful in traversing the current topography of data in schools.

Hot Tip: Learn how schools are organized. This reveals how different programs (and data) are situated within the organization. Albeit oversimplified, schools can be divided loosely by what all students v. some students get and what is directly related to academics v. what is not. The first distinction helps us understand general and special education, the second divides the three R’s of education from health, mental health, and social service supports. Considerable data collected in schools today are used to identify who (all or some) should get what resources and, ideally, if those resources are effective.

Hot Tip: Learn about major initiatives churning up data in our nation’s schools (e.g., Positive Behavior Supports, PBS; http://www.pbis.org/; and Response to Intervention, RtI, http://www.rti4success.org/).  Focused on behavior and academics, respectively, PBS and RtI use the public health model to organize school-wide systems of tiered supports that use data to drive resource allocation. Both have computer-based systems to help schools collect and use data (SWIS, School-wide Information System, http://www.swis.org/; AIMSweb, http://www.aimsweb.com/).

Hot Tip: All schools collect student data. Data quality and organization may not be ideal, but all schools have data on academic achievement, attendance, free-and-reduced lunch, suspensions/expulsions, and graduation. Regardless of relative value to an evaluation, school personnel use considerable resources to collect data and it’s important to acknowledge these efforts. This is also a terrific and relatively easy place to build their evaluation capacity.

Hot Tip: Talk to school counselors, nurses, and social workers. They are collecting some kind of data and, because they are more likely to have training in some evaluation-related area, they can be valuable local (i.e., sustainable) resources. School counselors are increasingly called on to evaluate their guidance programs, and school nurses may use data collection resources associated with Coordinated School Health Program model (CSHP; http://www.cdc.gov/HealthyYouth/CSHP/index.htm).

Hot Tip: Ask for data, any data. See if schools are involved in any research, have informally collected feedback/satisfaction data, or have been tapped to participate in regional or national surveys (e.g., Youth Risk Behavior Surveillance Survey, PRIDE). Data access and quality will vary, but this is great information about a school’s previous evaluation experiences (good and bad). This is also an entry point to help schools advocate for themselves when folks come asking for data in the future.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

My name is Mario Lurig, and I am the Technical Training Manager at SurveyGizmo, leading documentation, webinars, and other methods for educating evaluators on using our online survey tool.

Regardless of how you collect evaluation data, the data collected requires protection and careful handling to ensure that the results from your respondents are kept in strict confidence. Below I’ve compiled some hot tips for properly sharing this data with 3rd parties, as well as safely managing the data throughout the analysis period. Special thanks go to Marni Zapin, SurveyGizmo’s Product Marketing Manager, for assisting in compiling these tips.

Hot Tip: Store your data securely

If collecting paper surveys, once the data has been entered into a spreadsheet or database, the original surveys should be shredded and destroyed. However, protecting your digital files is just as critical. If storing them locally on your office computer, ensure the files are either encrypted or protected by a system password. Consider using a shared drive that includes both of the previously mentioned security measures. If the data is stored in an online tool, make sure the privacy policy specifies that the data will not be shared or used by the provider.

Hot Tip: Share with your clients only relevant data in compiled reports

When using online tools for collected data, many come with built-in methods for compiling that data in formatted reports that can be shared with your clients. To ensure the data is safe, distribute the report to your clients with a shared web link that is password protected (see a theme?), can be disabled at anytime, does not require granting access to your account, or all three! Secondly, make sure you use segments or filters to only share the data that is relevant to the client.

Hot Tip: Remove sensitive fields

Personal and demographic information can be considered sensitive data, which will be no surprise to evaluators. However, sometimes after compiling data segments based on the sensitive data, the results are compiled without removing these sensitive fields. In most cases, the sensitive data does not need to be reported directly, but only used to identify a group. Remove these fields from your results!

Hot Tip: Reassure your respondents about the data’s safety

A common incentive is to share your results with the respondents at the conclusion of the study. When you send a follow-up email, make sure to include a statement that reassures your respondents about the safety of their data and the results. This simple act will reassure everyone from your sources to your clients, while ensuring a long relationship with both.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hello, my name is Lisa Garbrecht. As a Research Associate at EVALCORP Research & Consulting, I work on numerous projects requiring data collection from youth. As you may know, it is not always easy to obtain high quality data (i.e., sufficient numbers of completed surveys, academic data, etc.) when relying on schools to help facilitate the data collection process. Below are a few tips that have proved useful!

Hot Tip #1:  Take time up front to identify the right data and methods. With the limited time and resources faced by schools and school-based programs today, it is important to collaborate with clients early on to identify priority needs and ensure that data are collected efficiently. Would a brief post-survey suffice instead of a comprehensive pre-post? Be strategic and include only items that really matter. Phrase items clearly and simply to ensure they are easily understood. Show schools you respect their time by only asking for the most vital information to inform the evaluation. 

Hot Tip #2:  Partnerships are key. Working together, evaluators and clients can build mutually beneficial relationships with schools to overcome their resistance to providing data. By showing school personnel and stakeholders how the findings may be of use and providing them with the necessary tools and databases, schools are more willing to collect and provide data in a timely manner. Communicate regularly with clients and schools, providing contact information so that you can answer their questions and offer assistance as needed.

Hot Tip #3:  Look at the data before it is too late. Whenever possible, do not wait until the end of the data collection process to analyze what’s coming in. Running the data early on allows you to identify problems with the tool or data collection process and make changes. Monitor data quality on at least a quarterly basis. This allows you to provide clients and schools with formative information that can serve to strengthen their programs and their motivation for assisting with ongoing data collection.

Hot Tip #4:  A little incentive goes a long way. Use incentives with project staff, school personnel and/or students as allowed. For instance, EVALCORP uses an award system for rewarding site staff members who consistently collect accurate, legible and complete survey data with a small gift card and certificate of appreciation. Pizza parties or other youth-friendly activities are other alternatives for showing clients/schools your thanks. If tangible incentives are not possible, be sure to let those involved know the value of their input and how much you appreciate their time. Oftentimes, a “Thank you and I really appreciate your help” goes a long, long way!

This aea365 Tip-a-Day contribution comes from the American Evaluation Association. If you want to learn more from Lisa, check out the sessions sponsored by the PreK-12 Educational Evaluation TIG on the program for Evaluation 2010, November 10-13 in San Antonio. If you would like to contribute an aea365 Tip, please send an email to aea365@eval.org.

· ·

Greetings from Columbia, SC! My name is Heather Bennett, MSW, and I have experience working in the field of evaluation for both the public and private sector. Currently, I work as a Research Associate in the Office of Program Evaluation (OPE) at the University of South Carolina where I have the opportunity to lead and work collaboratively on state and federally funded education initiatives in South Carolina. One of my primary responsibilities is to lead our qualitative data analysis efforts, including the analysis of video or audio recordings of cognitive labs, focus groups, interviews, and responses to open-ended survey items.

Lesson Learned: For my tip-a-day for aea365 I am going to focus on one vital and fundamental lesson I’ve learned through the analysis of responses to open-ended survey items — the quality of the question asked has the greatest impact on the data analysis process.

As evaluators, I’m sure we have all inherited some projects with the corresponding data collection instruments and noticed some issues with the construction of items…or worse, we have looked back on the open-ended items we’ve developed and asked ourselves: “What was I thinking?” Upon inheriting the evaluation of a program, I was soon reminded of the impact item writing can have on data management. Issues of data utility arose as my team and I reviewed the structure of qualitative items and worked to develop clear coding structures for corresponding data.

Hot Tip: Poorly written items do not always follow the “garbage in, garbage out” scenario. However, it takes more time to take-out the trash and get to meaningful data (data cleaning, analysis, coding) when you start with bad items. Below are a few things to keep in mind when developing open-ended items that will support your analysis and coding efforts once the data is collected.

First, you must have a clear understanding of what it is you want to learn about the project before you do anything else. What information do you really hope to gain? What is its utility for the program? This process should be guided by the project scope and involve project stakeholders to ensure the usefulness of the data collected.

Now that you have focused your data collection efforts, use these tips when developing your open-ended item(s):

  1. Ask one question at a time.
  2. Avoid leading questions.
  3. Avoid including personal biases in questions.
  4. Be specific about the topic.
  5. DO NOT ask questions that can be answered with yes/no.
  6. Indicate the number of responses requested from the participant.
  7. Ask clear and concise questions to avoid participant fatigue.

Following these tips will serve to improve your efforts in collecting focused and clear information from program participants.

This aea365 Tip-a-Day contribution comes from the American Evaluation Association. If you want to learn more from Heather, join us in San Antonio this November for Evaluation 2010 and check out her session on the Conference Program. If you would like to contribute an aea365 Tip, please send an email to aea365@eval.org.

· ·

Hi, I’m Dawn Hanson Smart, Senior Associate at Clegg & Associates, Inc. in Seattle, a consulting firm focused on planning and evaluation with nonprofits, state and local governments, tribal organizations, and foundations. I’ve been an evaluator for more than 25 years, with an eclectic mix of clients and projects. I think it is this diversity of fields and topics that keeps me interested and excited about my work. It also leads me to explore a wide range of reading material. I recently finished The Immortal Life of Henrietta Lacks by Rebecca Skloot, the story of the woman whose cells became the first to be grown in a laboratory and the many related stories of how it came about and the impact on her family and on research science.

Lesson Learned: Beyond the admiration we may feel for their work, these kinds of authors can stimulate our thinking, bringing new perspectives to the way we approach our own. Rebecca Skloot’s persistence over more than ten years to gather the data she needed, her diligence in the face of barriers, and her unfailing respect and caring for the people she worked with inspired me. The book made me think about how I might adjust my approach to (and my attitude about) a difficult evaluation I was conducting. It also made me remember the encouragement and enlightenment provided by books in the past. So I challenge you to broaden your reading habits and explore.

Rad Resource: Great places to get book ideas and see reviews include the New York Times Sunday Book Review and the New Yorker’s, Books Briefly Noted.

Rad Resource: TED is a nonprofit “devoted to Ideas Worth Spreading.” TED’s annual conference brings together speakers about science, business, the arts and global issues. Over the conference’s four days, 50 speakers each get an 18-minute time slot to share their knowledge and ideas and to inspire the audience with their creativity and passion for their work. You can see video clips of conference speakers at http://www.ted.com/ and identify individuals and topics you’d like to know more about.

Hot Tip: Book clubs are terrific places to learn about books you might not otherwise select for yourself. If you’re not a part of a club, join one. If you’re in one that’s beginning to feel stale, seek out a new one that includes nonfiction, fiction, history, biography, and poetry books. Or join a club online. The Salon Reading Club offers a stimulating community where you can get together with others and discuss books. Or, try the New Yorker’s Book Club.

So, explore — read outside your field something in an area you’ve never even considered of interest. You might be surprised!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

· · ·

My name is Susan Kistler, AEA’s Executive Director, and I contribute each Saturday’s aea365 post. Earlier this week, I had the pleasure of speaking about the democratization of data at the AEA/CDC Summer Evaluation Institute.  While a candidate, US President Barak Obama committed to putting government data online in universally accessible formats (see here). At a 2009 TED Talk, Tim Berners-Lee – the founder of the Internet – urged the audience to share “Raw Data Now” (see here). And just this year, statistician and activist Hans Rosling noted that times have changed and we need to get data to the public (see here). Taken together, the message is clear – we’re in a new age of data sharing. But what does this mean for evaluation?

Lessons Learned for Evaluators:

  • There are competing forces at work, with competing expectations and needs. As an example, the call for open data sharing can be in conflict with the Health Information Privacy Act (HIPPA) which protects individual privacy.
  • We may need to bring new people on our evaluation teams in order to create mechanisms for the public to access and use data in ways that promote utility and understanding. One strong example comes from the Kids Count Data Center, an initiative of the Annie E Casey Foundation that makes measures of child well being accessible to the public.
  • The US Government is making strides towards realizing the Obama vision for online data sharing via such initiatives as data.gov – but weaker progress in terms of making data available in universally accessible formats or in ways that are understandable not only to researchers but also to the general public.
  • Making data publicly available can result in intriguing use such as the recent competition from The Sunlight Foundation that provided a $5,000 prize from the US Department of Health and Human Services for “innovative applications that improve the public’s understanding of community health performance” based on a DHHS dataset. Type your US postal code into the winner “County Sin Rankings” for a potentially surprising look at your locale.
  • Interactive interfaces and tools that allow the general public to analyze data and create a near-infinite number of data visualizations, can produce results with subtleties that may not be apparent to the general consumer. Take a look at these two maps of infant mortality rates from the New York Times Data Visualization Lab, comparing rates from 1960 and 2004 in selected countries. You’ll need to look carefully at the scales to see the considerable difference.

Rad Resource: This downloadable session handout provides links to 20 tools for making data publicly accessible – from examples of data in use to data repositories to free tools for collecting, analyzing, and visualizing data.

The above opinions are my own and do not necessarily reflect those of the American Evaluation Association.

·

<< Latest posts

Older posts >>

Archives

To top