AEA365 | A Tip-a-Day by and for Evaluators

CAT | Quantitative Methods: Theory and Design

Hi there – my name is Jennifer Catrambone and I am the Director of Evaluation & Quality Improvement at the Ruth M Rothstein CORE Center in Chicago, Illinois. That’s an Infectious Disease Clinic specializing in HIV/AIDS. I’m presenting on my favorite nerdy topic – the what and how of Nonparametric Statistics. I’ve taught both parametric and nonparametric stats at the graduate and undergraduate levels and have done stats consulting. Hang on!! Before you go running away because I used the word Statistics a bunch of times already, let me get a couple more lines out.

It hurts my soul (not like sick puppies or mullets, but still…) when people just reach for the parametric stats, e.g., ANOVAs, T Tests, etc…, without thinking carefully about whether those are the best ones for their data. Why? Because those tests, the parametric ones that we all spent all that time learning in school, are sometimes wildly inappropriate and using them with certain very common kinds of data actually decreases your likelihood of finding that sought-after p<.05. The trick is to match your data set, with its imperfections or unpredictable outliers, to the right kind of stats.

Lesson Learned: So, what situations require nonparametric statistics? They can be broken down into a few major categories:

  1. The data set is very small. Sometimes that N just does not get to where we want it to be.
  2. The subgroups are uneven. Perhaps there are many pretests and very few post tests, or maybe you let people self-select which group they were in and no one chose the scary sounding one.
  3. The data is very skewed. Bell Curve, Schmell Curve.
  4. Your variables are categorical or ordinal.

There aren’t a lot of resources on Nonparametric Statistics out there. College/grad school statistics textbooks offer minimal information on nonparametric stats, focusing disproportionately on Chi Squares but rarely include info on the post hoc tests that should follow that test. One excellent Nonparametric Stats resource, though published in 1997, is by Marjorie Pett and is entitled, “Nonparametric Statistics for Health Care Research.” The popular stats texts by Gravetter and Wallnau have also introduced decision trees for nonparametric stats that are incredibly useful for determining what test to use.

OK – so all of that being said, the bad news is that many of us just use Parametric Stats because that’s what we know, regardless of the data, and accept that with our messy data, effects will be harder to come by. The great news is that that’s not necessary. Nonparametrics take all that into account and slightly modifies parametric tests (e.g., using medians instead of means), making it so that things like skew and tiny samples are not effect-hiding problems anymore.

Want to learn more? Register for Nonparametric Statistics: What to Do When Your Data Breaks the Rules at Evaluation 2015 in Chicago, IL.

This week, we’re featuring posts by people who will be presenting Professional Development workshops at Evaluation 2015 in Chicago, IL. Click here for a complete listing of Professional Development workshops offered at Evaluation 2015. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Greetings! I am Linda Cabral from the University of Massachusetts Medical School’s Center for Health Policy and Research. A big part of my job uses qualitative methods to evaluate different health and human services programs. Our data collection processes can include utilizing one-on-one or group interviews and as well as focus groups. With this type of narrative (i.e., first person data collection), decisions must be made up front as to the ultimate format of your data set. One of the decisions is whether or not to audio record these data collection events and another is whether these audio files will be transcribed. Transcribing data can be a tedious process requiring several hours for each recorded interview. A general rule is that the text of a 30-40 minute interview takes about 1-2 hours to type and results in about 15-20 pages of text. Recently, we have been faced with a myriad of projects requiring decisions on how formal our transcription process should be. Let me offer you some of our thinking and lessons learned!

Lessons Learned:

  • Decisions are needed as to the level of detail needed from each qualitative data collection event, which can range from a verbatim transcript to a less formal write-up of notes. While transcribing your own data can have significant analytic benefits (getting close and personal with the material), it may not be practical for everyone – particularly if you’re time-strapped.
  • Transcription of interviews allows for each evaluation team member to go through the transcript carefully, providing an easily readable document from the study. Having a transcript can facilitate working together in a team where the tasks have to be shared. Agreement about data interpretation is key.
  • When considering outsourcing transcription:
    • Realize that a fully transcribed interview will result in pages and pages of data to sift through. There will be a lot of “noise” in there that could potentially be eliminated if the transcription was done in-house by evaluators familiar with the project’s evaluation aims.
    • You have choices as to the type of transcript that would be most helpful to you, including: word-by-word verbatim; clean verbatim (removing ‘hmm’ and ‘you know’); or one with improved grammar and readability.
    • You have options ranging from independent contractors to large firms that specialize in transcription services. Transcribers can be paid by the word, the hour, or the length of time of the recording.

Hot Tips:

  • Always have your evaluation aims drive your decision about whether to transcribe or not.
  • Plan ahead for how notes, audio recordings, and transcripts will be stored and how personal identifiers will be used in order to main data integrity.
  • Budget the necessary time and resources up front whatever your decision is!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi, we are Pei-Pei Lei and Carla Hillerns from the Office of Survey Research at the University of Massachusetts Medical School. Have you ever selected an answer to a survey question without reading all of the choices? Maybe you paid more attention to the first choice than the rest? Today, we’d like to share a technique that helps to minimize the impact of these types of scenarios – randomized ordering of survey response options.

Randomizing the order of response options may improve data quality by reducing the order effect in your survey. When there is a list of response options, respondents often have a tendency of selecting the most prominent. For example, in a paper survey, the first option may be most apparent. In a phone survey, the last option may be most memorable. If implementing an online survey, there may be a tendency to choose from the middle of a long list – because the center is more prominent.

By randomizing the order, all options have the same possibility of appearing in each response position. In Example A below, “Direct mail” appears in the top spot. However, in Example B, the responses have been randomly reassigned and “Television” now appears at the top.

Lei

Hot Tips:

  • Do not randomize the order if the response options are better suited to a pre-determined sequence, such as months of the year or alphabetization, or if using a validated instrument that needs to maintain the full survey as developed.
  • If the response list is divided into sub-categories, you can randomize the category order as well as the items within each category.
  • If your list includes “Other (Please specify: __________)” or “None of the above”, keep these at the bottom so the question makes sense!
  • If using the same set of response options for multiple questions, apply the first randomized ordering to the subsequent questions to avoid confusion.
  • Randomization is not a cure for all questionnaire design challenges. For example, respondents probably won’t pay as much attention to each response option if the list is extremely long or the options are excessively wordy. So be reasonable in your design.

Lesson Learned: It’s easy to administer randomization in web and telephone surveys if your survey platform supports this function. A mail survey will require multiple versions of the questionnaire. You’ll also need to account for these multiple versions as part of the data entry process to ensure that responses are coded accurately. 

Rad Resources:

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

This is Josh Twomey again from UMass Medical School’s Center for Health Policy and Research. As promised in yesterday’s Part 1 posting, I wanted to walk through an example demonstrating aspects of Bayesian analysis that evaluators might find advantageous.

Part 1 mentioned the use of the Bayes Factor (BF) as a decision-making tool. BFs tell us the degree one hypothesis is supported by the data in relation to how much another hypothesis is supported by the data. Thus, a BF is an odds ratio showing us which hypothesis is more likely. Now, let’s look at that example.

Suppose you are evaluating the effectiveness of a health psychology program in helping patients manage chronic disease. As part of the evaluation, you measure the self-efficacy of 200 patients managing their diabetes before and after working with a health psychologist. Traditionally, you could do this with a paired t-test of patients’ pre and post self-efficacy scores. As good evaluators, we would begin this analysis with some idea (based on review of similar evaluations or literature) as to the effectiveness of our program. However, in a traditional test, this prior knowledge cannot be factored into our analysis. Our test would produce t and p values such as t = – 3.19, p < .05. With this result we can state post scores are significantly higher than pre scores, but we cannot state the extent to which this conclusion is more likely than our null hypothesis.

With Bayesian analysis, we conduct this paired sample t-test but weight our data by our prior knowledge. For example, if past evaluations tell us that we should expect small effect sizes, we can specify a prior whereby small effect sizes are more likely than larger ones. In this Bayesian framework, a t = -3.19 corresponds to a BF of 10.1. This means that our hypothesis that post scores are different than pre scores is 10.1 times more likely than a hypothesis of no difference.

Hot Tips: The example above highlights 3 advantages of Bayesian analysis:

  • Prior knowledge is incorporated into the analysis as our data is weighted by this knowledge;
  • We are given direct odds associated with our conclusion; and
  • The interpretation of the BF is a clear, intuitive interpretation of our results for stakeholders to understand.

Lessons Learned: Over the course of my journey, which is far from over, I have learned that Bayesian analysis is complex – full of intimidating terms such as conjugate priors and Markov Chains. But considering its advantages, as well as growing demand in our field, I have found the journey to be well worth it.

Rad Resources: Bayesian Factor calculators and literature can be found at: pcl.missouri.edu.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

My name is Josh Twomey, Assistant Professor of Family Medicine & Community Health, at UMass Medical School’s Center for Health Policy and Research. Perhaps you have noticed the term ‘Bayesian’ popping up now and then in the evaluators’ break room. I certainly have, and in recent months, set out on a statistical journey to find out why. In this two-part entry, I would like to share some discoveries of this journey. Part 1 includes a basic overview of Bayesian analysis. Part 2 (coming tomorrow) gives an example of how this may benefit our work as evaluators. One note of caution: This may seem less relevant to those doing strictly qualitative data collection and reporting!

Most of us are trained in Null Hypothesis Significance Testing (NHST) whereby we reject a null hypothesis (p < .05) or fail to reject the null (p > .05). This decision is justified by the data we collect, but does not take into account past research findings or expert opinion. Bayesian analysis differs from NHST in that past knowledge is included in our analysis, thereby having a direct influence on our conclusions.

The first step of Bayesian analysis is to quantify this past knowledge via a prior distribution or prior for short. Using priors we are able to specify distribution(s) of parameters (e.g., means, standard deviations) we are interested in. In cases where we have a lot of prior information, we may set up narrow distributions (informative priors). When we do not have a lot of prior information, we may set up very wide distributions (noninformative priors). Priors are used to weight the likelihood of our collected data to produce the posterior distribution. Thus, the posterior is a result of past knowledge updated by our collected data. It is from this posterior where samples can be drawn and conclusions about our evaluations are made.

Hot Tips: In addition to allowing for the use of past knowledge, advantages of Bayesian statistics include:

  • Decision-making tools such as Bayes Factors and Highest Density Intervals (HDIs), which can be easier for stakeholders to understand compared to p values and confidence intervals.
  • No need to limit the number of hypotheses you wish to test with your data for fear of inflated Type I error (which in my experience can frustrate stakeholders).
  • Better capacity to work with Ns that are small, limiting our ability to detect differences/trends, or large where differences may be detected due to large samples, not meaningful differences.

Rad Resources:

Bayesian Statistics for the Social Sciences by David Kaplan

Doing Bayesian Data Analysis by John Kruschke

For quick reference/definitions of NHST, p-values, and Bayesian inference click HERE.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Did we get your attention? We hope so. We are Carla Hillerns and Pei-Pei Lei – survey enthusiasts at the Office of Survey Research at the University of Massachusetts Medical School.

An email subject line can be a powerful first impression of an online survey. It has the potential to convince someone to open your email and take your survey. Or it can be dismissed as unimportant or irrelevant. Today’s post offers ideas for creating subject lines that maximize email open rates and survey completion rates.

Hot Tips:

  • Make it compelling – Include persuasive phrasing suited for your target recipients, such as “make your opinion count” and “brief survey.” Research in the marketing world shows that words that convey importance, like “urgent,” can lead to higher open rates.
  • Be clear – Use words that are specific and recognizable to recipients. Mention elements of the study name if they will resonate with respondents but beware of cryptic study names – just because you know what it means doesn’t mean that they will.
  • Keep it short – Many email systems, particularly on mobile devices, display a limited number of characters in the subject line. So don’t exceed 50 characters.
  • Mix it up – Vary your subject line if you are sending multiple emails to the same recipient.
  • Avoid words like “Free Gift” (even if you offer one) – Certain words may cause your email to be labeled as spam.
  • Test it – Get feedback from stakeholders before you finalize the subject line. To go one step further, consider randomly assigning different subject lines to pilot groups to see if there’s a difference in open rates or survey completion rates.

Cool Trick:

  • Personalization – Some survey software systems allow you to merge customized/personalized information into the subject line, such as “Rate your experience with [Medical Practice Name].”

Lesson Learned:

  • Plan ahead for compliance – Make sure that any recruitment materials and procedures follow applicable regulations and receive Institutional Review Board approval if necessary.

Rad Resource:

  • This link provides a list of spam trigger words to avoid.

We’re interested in your suggestions. Please leave a comment if you have a subject line idea that you’d like to share.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hi! We are M. H. Clark and Haiyan Bai from the University of Central Florida in Orlando, Florida. Over the last several years propensity score adjustments (PSAs) have become increasingly popular; however, many evaluators are unsure of when to use them. A propensity score is the predicted probability of a participant selecting into a treatment program based on several covariates. Theses scores are used to make statistical adjustments (i.e., matching, weighting, stratification) to data from quasi-experiments to reduce selection bias.

Lesson Learned:

PSAs are not the magic bullet we had hoped they would be. Never underestimate the importance of a good design. Many researchers assume that they can fix poor designs with statistical adjustments (either with individual covariates or propensity scores). However, if you are able to randomly assign participants to treatment conditions or test several variations of your intervention, try that first. Propensity scores are meant to reduce selection bias due to non-random assignment, but can only do so much.

Hot Tip:

Plan ahead! If you know that you cannot randomly assign participants to conditions and you MUST use a quasi-experiment with propensity score adjustments, be sure that you measure covariates (individual characteristics) that are related to both the dependent variable and treatment choice. Ideally, you want to include all variables in your propensity score model that may contribute to selection bias. Many evaluators consider propensity score adjustments after they have collected data and cannot account for some critical factors that cause selection bias. In which case, treatment effects may still be biased even after PSAs.

Hot Tip:

Consider whether or not you need propensity scores to make your adjustments. If participants did not self-select into a treatment program, but were placed there because they met a certain criterion (i.e., having a test score above the 80th percentile), a traditional analysis of covariance used with regression discontinuity designs may be more efficient than PSAs. Likewise, if your participants are randomly assigned by pre-existing groups (like classrooms) using a mixed-model analysis of variance might be preferable.  On the other hand, sometimes random assignment does not achieve its goal in balancing all covariates between groups. If you find that the parameters of some of your covariates (i.e., average age) are different in each treatment condition even after randomly assigning your participants, PSAs may be a useful way of achieving the balance random assignment failed to provide.

Rad Resource:

William Holmes recently published a great introduction to using propensity scores and Haiyan Bai and Pan Wei have a book that will be published next year.

Want to learn more? Register for Propensity Score Matching: Theories and Applications at Evaluation 2014.

This week, we’re featuring posts by people who will be presenting Professional Development workshops at Evaluation 2014 in Denver, CO. Click here for a complete listing of Professional Development workshops offered at Evaluation 2014. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Hello! My name is Amber Hill and I am a research specialist at McREL International’s Denver, Colorado office. My work focuses on education research and my responsibilities include managing online surveys administered to state departments of education, districts, school staff, parents, and students in the United States, Pacific region, and Australia. Encouraging online survey participation can be tricky, which is why I use a variety of methods.

Hot Tip – Work with the IT Pros

Ensuring that participants receive the survey in the first place can be half of the battle. No matter your level of information technology (IT) expertise, it is helpful to coordinate efforts between the IT pros who work for your survey software provider, your own organization, and the organization for which you are administering the survey. Those three groups can help you with white listing, test emails, firewalls, and broken links.

Hot Tip – Communicate Early, Communicate Often

Participants are often leery of participating in a survey administered by a stranger, especially if the content is sensitive. Working with a partner organization that is familiar to participants helps increase understanding about the purpose, value, and trustworthiness of the survey and evaluator. Partner organizations may send an e-mail to participants with the evaluator’s name and contact information in advance of the recruitment e-mail. Follow-up and reminder e-mails from the evaluator that includes references to the partner organization shows participants the coordination between the organizations. Keeping surveys open for extended amounts of time also allows for more reminders and opportunities for participants to ask questions.

Cool Trick- Provide Participate Appropriate Incentives

Incentives such as monetary compensation or prizes can motivate participants to spend their time on the survey. Try to think of something that participants would genuinely enjoy or find useful. Incentives may go to participants or survey administrators, depending on how the survey is distributed. When funding is limited, a drawing for a prize among participants who elect to provide their contact information may be effective.

Rad Resources – Denver’s Urban Trails and Parks

While at Evaluation 2014, you will notice that Denver’s outdoor culture thrives everywhere from mountains peaks to downtown. The Colorado Convention Center bumps up against the Cherry Creek Trail, which if taken north leads to Confluence Park and south leads to Sunken Gardens Park and beyond. A quick exploration west will hook up with the South Platte River Trail and to Sloan’s Lake Park.  Longer treks east of downtown will reward visitors with mountain views at Cheesman Park (go to the Pavilion) and animal life at the Denver Zoo and City Park. Get outside!

We’re thinking forward to October and the Evaluation 2014 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Registration will soon be open! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

Hi!  This is Andrea Crews-Brown, Tom McKlin, and Brandi Villa with SageFox Consulting Group, a privately owned evaluation firm with offices in Amherst, MA and Atlanta, GA, and Shelly Engelman with the School District of Philadelphia. Today we’d like to share results of a recent survey analysis.

Lessons Learned: Retrospective vs. Traditional Surveying

Evaluators typically implement pre/post surveys to assess the impact a particular program had on its participants. Often, however, pre/post surveys are plagued by multiple challenges:

  1. Participants have little knowledge of the program content and thus leave many items blank.
  2. Participants complete the “pre” survey but do not submit a “post” survey; therefore, it cannot be used for comparison.
  3. Participants’ internal frames of reference change between the pre and post administrations of the survey due to the influence of the intervention. This is often called “response-shift bias.” Howard and colleagues (1979) consistently found that the intervention directly affects the self-report metric between the pre-intervention administration of the instrument and the post-intervention administration.

Retrospective surveys ask participants to compare their attitudes before the program to their attitudes at the end. The retrospective survey addresses most of the challenges that plague traditional pre/post surveys:

  1. Since the survey occurs after the course, participants are more likely to understand the survey items and, therefore, provide more accurate and consistent responses.
  2. Participants can reflect on their growth over time, giving them a more accurate view of their progression.
  3. Participants will take the survey in one sitting which means that the response are more likely to be paired.

Lesson Learned: Response Differences

To analyze response-shift bias, we compared the pre responses on traditional pre/post items measuring confidence to “pre” responses on identical items administered retrospectively on a post survey. When asked about their confidence at the beginning of the course, a mean of 4.47 was reported while on the retrospective survey a value of 3.86 was reported. The students expressed significantly less confidence on the retrospective. A Wilcoxon Signed-Rank Test was used to evaluate the difference in score reporting from traditional pre to retrospective pre. A statistically significant difference (p < .01) was found indicating that the course may have encouraged participants to recalibrate their perceptions of their own confidence.

McKlin

Rad Resource:  Howard has written several great articles on response-shift bias!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

· · ·

Hi, I’m Harlan Luxenberg from Professional Data Analysts, Inc., a public he alth evaluation firm in Minneapolis, and I’d like to share some thoughts about certain situations where databases may be more useful than Microsoft Excel.  Excel is great for quickly crunching data and managing small datasets; however, using Excel in the wrong situations can actually make your data management tasks trickier.

Below are problems that our colleagues have encountered in Excel and reasons why we think that databases would be better solutions in these cases.

Hot Tip: Know which situations to use a database.

Luxenberg

Rad Resource: Anyone can learn databases!

While the thought of learning databases can sound intimidating, anyone can learn them and there are tons of resources that will help you get going! There are numerous blogs and websites, and even free online classes such as Coursera.

Rad Resource: Start with Microsoft Access

A good database to start with is Microsoft Access (http://office.microsoft.com/en-us/access/), which is part of the Microsoft Office Professional suite, and which may already be installed on your computer. Microsoft Access allows users to build reports, create data collection forms, visually create tables, and integrates seamlessly with Excel (which you can still use to create beautiful charts).

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

<< Latest posts

Older posts >>

Archives

To top