AEA365 | A Tip-a-Day by and for Evaluators

TAG | statistics

Greetings AEA community, I’m Pei-Pei Lei, a biostatistician in the Office of Survey Research at the University of Massachusetts Medical School. Have you been looking to expand your skill set in statistical programming? Have you wondered if R is the appropriate statistical software package for your needs? The purpose of this post is to help you decide whether R is right for you and, if so, how you can get started using it.

R may be the right tool if you:

  • Need to manage and/or analyze quantitative data
  • Are looking for a free alternative to commercial software packages, such as SAS, SPSS, and STATA
  • Don’t mind writing computer code – does print (“Hello, world!”) look easy enough to you?
  • Want to create nice-looking and informative figures and graphics (see this website for example)

If you’re not sure, here are some places for you to get a feel for R language:

  • TryR: This website provides online interactive step-by-step practice on the webpage
  • DataCamp: This website provides online interactive step-by-step practice (more material than TryR)

Hot Tips:

The following is a list of MOOCs (Massive Open Online Courses) that can help you learn R for free (or pay a fee for a verified certificate):

  • R programming on Coursera: It’s a 4-week course to go through basic R programming knowledge. It provides a weekly quiz and a final project for you to test your skills. Good for beginning to intermediate users.
  • Introduction to R for Data Science on edX: It’s a self-paced 4-week course to go through basic R programming knowledge. This course is using DataCamp for class materials and exercises. Good for beginners.
  • R Basics – R Programming Language Introduction on Udemy: This is a self-paced course that goes through basic set up such as downloading the software and coding. Good for beginners.
  • Data Analysis with R on Udacity: This course takes about 2 months to finish (it’s also part of the Data Analyst nanodegree program). Its tutorial videos show coding processes in RStudio. Good for beginning to intermediate users.

You can also install the Swirl R package to learn R in R. It gives you interactive instructions for different topics. This is good for intermediate users.

Rad Resources:

  • R-bloggers: This is a repository of R-related articles, including tutorials. You can subscribe to the mailing list to receive the latest articles.
  • Stack overflow: This is a forum where you can post your question and get answers, or even better, provide answer to others’ questions!

Lessons Learned:

Don’t be intimidated by the many choices you have in learning R. They are the means to reach your goal. So pick one that you like and get started!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hi there – my name is Jennifer Catrambone and I am the Director of Evaluation & Quality Improvement at the Ruth M Rothstein CORE Center in Chicago, Illinois. That’s an Infectious Disease Clinic specializing in HIV/AIDS. I’m presenting on my favorite nerdy topic – the what and how of Nonparametric Statistics. I’ve taught both parametric and nonparametric stats at the graduate and undergraduate levels and have done stats consulting. Hang on!! Before you go running away because I used the word Statistics a bunch of times already, let me get a couple more lines out.

It hurts my soul (not like sick puppies or mullets, but still…) when people just reach for the parametric stats, e.g., ANOVAs, T Tests, etc…, without thinking carefully about whether those are the best ones for their data. Why? Because those tests, the parametric ones that we all spent all that time learning in school, are sometimes wildly inappropriate and using them with certain very common kinds of data actually decreases your likelihood of finding that sought-after p<.05. The trick is to match your data set, with its imperfections or unpredictable outliers, to the right kind of stats.

Lesson Learned: So, what situations require nonparametric statistics? They can be broken down into a few major categories:

  1. The data set is very small. Sometimes that N just does not get to where we want it to be.
  2. The subgroups are uneven. Perhaps there are many pretests and very few post tests, or maybe you let people self-select which group they were in and no one chose the scary sounding one.
  3. The data is very skewed. Bell Curve, Schmell Curve.
  4. Your variables are categorical or ordinal.

There aren’t a lot of resources on Nonparametric Statistics out there. College/grad school statistics textbooks offer minimal information on nonparametric stats, focusing disproportionately on Chi Squares but rarely include info on the post hoc tests that should follow that test. One excellent Nonparametric Stats resource, though published in 1997, is by Marjorie Pett and is entitled, “Nonparametric Statistics for Health Care Research.” The popular stats texts by Gravetter and Wallnau have also introduced decision trees for nonparametric stats that are incredibly useful for determining what test to use.

OK – so all of that being said, the bad news is that many of us just use Parametric Stats because that’s what we know, regardless of the data, and accept that with our messy data, effects will be harder to come by. The great news is that that’s not necessary. Nonparametrics take all that into account and slightly modifies parametric tests (e.g., using medians instead of means), making it so that things like skew and tiny samples are not effect-hiding problems anymore.

Want to learn more? Register for Nonparametric Statistics: What to Do When Your Data Breaks the Rules at Evaluation 2015 in Chicago, IL.

This week, we’re featuring posts by people who will be presenting Professional Development workshops at Evaluation 2015 in Chicago, IL. Click here for a complete listing of Professional Development workshops offered at Evaluation 2015. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello! I’m Kathy McKnight, Principal Director of Research, Center for Educator Effectiveness at Pearson.

Today I completed my annual 2-day introductory workshop on Quantitative Methods, which I’ve offered at AEA’s annual conference every year since….well, I’ve lost track. Over the years, I’ve observed a lot of evaluators who participate in my workshop, hungry to learn something about statistics and quantitative methods.

Lessons Learned: A few observations to share: 1) It’s difficult for program evaluators to find quality workshops/educational opportunities for continuing their education in quantitative methods; I find this is the case for those at an introductory, intermediate, and advanced level, unless you’re located within a university (and even then, it’s not guaranteed you can find what you need). 2) I’m further convinced each year that training in statistics is not enough — evaluators need training in measurement and research methods/evaluation design as well. Without each of those critical elements, knowledge of any one of them alone is not sufficient. I’ve noticed that the greatest engagement in my workshop tends to be around methodological/philosophy of science issues with respect to how program evaluations are carried out, and what we can learn from them. Studying statistics helps bring out these issues: it’s not only about what tools are available, but how we can best use them, given our evaluation goals. These issues are what attracted me to program evaluation and keep me interested in this work. It seems to be the case for many others.

Hot Tips: For those interested in furthering their knowledge and skills in quantitative methods, AEA has a Quantitative TIG, and the good news is, we don’t bite! It’s a supportive, engaged group of individuals who share a strong interest in the methods by which we conduct evaluations, how we measure constructs we care about, and how we model relationships between those variables quantitatively. New members could help us identify ways to provide more and better training to our membership, and share resources. Additionally, AEA offers e-Studies (I offered one this past spring on basic inferential statistics) and “coffee break webinars” (brief presentations of a specific topic — I offered one on descriptive statistics). These are just a few of the online resources available to our membership*. The annual meeting also offers 1-day, 3-hour and 90-minute workshops, and a host of presentations focused on quantitative methods. These are well worth checking out as part of your continued education in the broad area of quantitative methods.

Rad Resource: Don’t forget your friend the internet — there are countless YouTube videos and statistics, measurement, and research methods websites that provide tutorials as well as a multitude of resources.

I wish you all a productive, educational conference this year in Washington DC! Please do check out the presentations from the Quantitative TIG.

*Coffee break webinars, e-Study workshops, and Professional Development workshops at the conference are paid content.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

I’m Jennifer Ann Morrow, a faculty member in Evaluation, Statistics, and Measurement at the University of Tennessee. I created a 12 Step process evaluators can follow to ensure their data is clean prior to conducting analyses.

Hot Tip: Evaluators should follow these 12 steps prior to conducting analyses for evaluation reports:

1. Create a data codebook

a. Datafile names, variable names and labels, value labels, citations for instrument sources, and a project diary

2. Create a data analysis plan

a. General instructions, list of datasets, evaluation questions, variables used, and specific analyses and visuals for each evaluation question

3. Perform initial frequencies – Round 1

a. Conduct frequency analyses on every variable

4. Check for coding mistakes

a. Use the frequencies from Step 3 to compare all values with what is in your codebook. Double check to make sure you have specified missing values

5. Modify and create variables

a. Reverse code (e.g., from 1 to 5 to 5 to 1) any variables that need it, recode any variable values to match your codebook, and create any new variables (e.g., total score) that you will use in future analyses

6. Frequencies and descriptives – Round 2

a. Rerun frequencies on every variable and conduct descriptives (e.g., mean, standard deviation, skewness, kurtosis) on every continuous variable

7. Search for outliers

a. Define what an outlying score is and then decide whether or not to delete, transform, or modify outliers

8. Assess for normality

a. Check to ensure that your values for skewness and kurtosis are not too high and then decide on whether or not to transform your variable, use a non-parametric equivalent, or modify your alpha level for your analysis

9. Dealing with missing data

a. Check for patterns of missing data and then decide if you are going to delete cases/variables or estimate missing data

10. Examine cell sample size

a. Check for equal sample sizes in your grouping variables

11. Frequencies and descriptives – The finale

a. Run your final versions of frequencies and descriptives

12. Assumption testing

a. Conduct the appropriate assumption analyses based on the specific inferential statistics that you will be conducting.

Lesson Learned: One statistics course is not enough. Utilize all the great resources that AEA offers to gain additional training in data analysis.

Rad Resources:

Want to learn more from Jennifer? Register for her upcoming AEA eStudy: The twelve steps of data cleaning: Strategies for dealing with dirty data and her workshop Twelve Steps of Data Cleaning: Strategies for Dealing with Dirty Evaluation Data at Evaluation 2013 in Washington, DC.

This week, we’re featuring posts by people who will be presenting Professional Development workshops at Evaluation 2013 in Washington, DC. Click here for a complete listing of Professional Development workshops offered at Evaluation 2013. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

Greetings, I am Cindy Weng, a bio-statistician II employed at Pediatrics Research Enterprise at Department of Pediatrics at the University of Utah. This post was written together with my colleagues Chris Barker, SWB project manager and Larry George, statistician at Problem Solving Tools.

I learned about this methodology through a project assigned by ASA Statistics Without Borders (SWB) in 2011. The goal of this project was to analyze under 5 years (U5) mortality of children before (“baseline”) and after (“endline”) humanitarian aid given at Afghan refugee Camps in Pakistan. Survival analysis was used to estimate the probability distribution of age at death from current status and admissible age-at-death data. Inadmissible ages at death placed the date of death after the survey dates!

The International Rescue Committee survey data contained inadmissible ages at deaths, so the Kaplan Meier nonparametric maximum likelihood estimator was, used along with estimators from current status data only.

Tips:

  • Maximum likelihood and least squares estimators differ. We estimated survivor functions from baseline and endline surveys. “MLE” and “LSE” denote maximum likelihood estimation and least squares estimates. They don’t always agree, because the methods are different approaches to estimation. In particular, LSE does not respond to noise. If noise is not uniform across the sample, LSE might be incorrect. The MLE takes noise into consideration. The MLE estimates in the figure are from current status data. They agreed pretty well with the Kaplan-Meier estimators from admissible ages at deaths.

Lessons learned:

  • Survey data is not always what is expected. Surveys should have cross-checking validation opportunities. Current status data provided the opportunity to make two estimates of survivor functions.
  • Expect unexpected outcomes. The baseline U5 estimates are over 10%, and the endline U5 estimate is approximately 4%. Pakistan’s country U5 is 8.7%. The endline U5 estimates standard deviation is less than 0.5%. The apparent reduction in U5 appears to be primarily a reduction in deaths after infant mortality in the first year. Infant mortality was almost 4% before and after.


Resources:

The American Evaluation Association is celebrating Statistics Without Borders Week. The contributions all week come from SWB members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

· · ·

Greetings I am Mark Griffin. At the time of writing this article I am fortunate enough to be in the middle of a world trip. Key events of my trip: last week I was in Fiji chairing the Pacific Conference for Statistics and Information Systems, my third trip to Fiji, with a rapidly developing workshop program that I have initiated. This week I am in Adelaide, Australia presenting lessons learnt in Fiji at the Australian Statistical Conference. Last night I held the first event for our societies’ section I founded earlier this year, Section for International Engagement. Tomorrow, I fly to North Korea to present Pyongyang University of Science and Technology and Statistics Without Borders co-organised event.

Working with friends and colleagues in developing nations is a true passion of mine. I have also set up an Australian NGO to deliver further training and consulting.

So what advice would I give to like-minded colleagues who have a similar passion?

Tips:

  • Find a mentor (or several). Working in developing countries is incredibly rewarding, but can also be incredibly demanding. Line up people who can support you through the emotional challenges involved, bounce ideas back and forwards, and celebrate with as you enjoy the fruits of your labour.
  • Make strong partnerships. The concept of partnership is a matter of humility, patience, and acceptance. As an outsider you might have superior academic knowledge, and yet your colleagues will best know what’s happening within their country, the needs and constraints, and will generally be the people who have made the largest personal commitment. Strong partnership requires constant communication back and forth about expectations, underlying motivation, and mutual appreciation.
  • Long-term sustainability is difficult. Many a kind-hearted person has gone in for a short duration and set up some potentially beneficial services (such as housing or healthcare facilities), and then quickly left again only for those services to fall into disuse. Any overseas colleague needs to think about the long-term benefits that collaboration will produce (and whether the benefits that you have in mind match the vision of the local people).
  • Communication, communication, communication. As a person who has recently gotten married I am constantly re-discovering the importance of improving all channels of communication. Constant communication is perhaps even more vital with colleagues living and working in completely different contexts. There are too many promising projects that have succeeded or failed, primarily due to the quality of the communication between the stakeholders.
  • Personal motivation is crucial. Make sure that a project is one that you personally are motivated about. At the end of the day, projects have joys and challenges, and to remain committed requires that you have personal motivation for the project to succeed.

Resources:

The American Evaluation Association is celebrating Statistics Without Borders Week. The contributions all week come from SWB members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

· · · ·

Greetings I am Mary Gray from the American University in Washington, D.C. and a member of Statistics Without Borders. Recently, I was involved in surveying Rwandan prisoners.

Lessons Learned:

  • Sample the appropriate stakeholders. Two years after the genocide that killed 800,000 Rwandans, primarily Tutsis, there were 80,00-90,000 imprisoned in a country of a few million and the prison population continued to grow by as many as 10,000 per month, the only release being death.  In spite of international horror over the brutal loss of life, international notions of justice demanded due process and some semblance of a speedy trial for the accused.  The post-genocide Rwandan government rightly claimed that the fragile judicial system, deprived of most of its personnel and much of its infrastructure, could not handle the prospective case load.  Donor governments who had already constructed several large new prisons asserted that however horrible the crimes of which they were accused it was not acceptable to put suspects in prison and throw away the key.  Why not, proposed representatives of the US and other nations, with the agreement of the Rwandan government, begin by selecting a sample of prisoners to bring to trial?
  • With large populations, stratify the sample. Because conditions under which large numbers of suspects were arrested and imprisoned  in different regions of the country, a stratified sample from four regions and the capital Kigali was used.
  • Prepare for the worst, records may be inadequate. By the time of the survey the prison conditions were generally adequate but records were difficult to acquire.  There were generally lists or card files that could be used for systematic sampling, but the information was generally limited.  Usually the crime was listed only as “genocide” without the names of victims or of arresting officers and no reference to the time and place of the offense.
  • Prepare for the worst, data may be missing. By the time of the survey little could be done about the missing data so expectations of what information could be gathered had to be revised.
  • Educate your stakeholders when possible. The lessons learned are the usual ones about educating those involved about the importance of considering what data needs to be collected. An unfortunate outcome was that those authorizing the survey did not understand that a random sample might not include those whom they were most eager to bring to trial.

Resource:

The American Evaluation Association is celebrating Statistics Without Borders Week. The contributions all week come from SWB members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

·

Greetings, I am Gary Shapiro, co-founder and current Chair of Statistics Without Borders (SWB). Recently, we started collaborating with the American Evaluation Association. This week aea365 will feature knowledge and resources from SWB members appropriate for professional evaluators.

Tips:

  • What is SWB? SWB is an  all-volunteer outreach group of the American Statistical Association. The group was formed in 2008 to provide pro bono statistical support to organizations, particularly from developing countries. Although the group started out small, it has quickly grown to involve over 500 volunteer statisticians around the world who provide pro bono consultancy activities.
  • Who can volunteer? SWB warmly welcomes volunteers from a wide range of backgrounds. We have members who are highly experienced statisticians, members who are new to the field of statistics but would like to work under the supervision of an experienced statistician, and members from other non-statistical disciplines (including the evaluation of community aid programs and data management).
  • How does one volunteer? Volunteering is simple, sign up online.
  • SWB projects – no job is too large or small. SWB has a broad scope of projects. SWB’s projects are the core of our mission.  Through these projects we help international health workers and others in resource-limited settings who do not have statistical training by partnering them with professional and student statisticians.  Some examples of projects include the design and analysis of epidemiological studies, the review of grant proposals for funding agencies in international health (health considered very broadly), and on-site training for current health projects or for the development of local staff. The scope of our work is diverse, ranging from survey design to analysis to efforts to provide statistical software for developing nations.

Resources:

  • Get help with a project. Do you know a group with limited resources? SWB is always looking to expand our list of collaborators and projects. If you are working in absolutely any field and would benefit from working with an expert group of statisticians, then we would love to hear from you.  If you have any ideas for projects that fall within the general scope of statistics we would love to talk further with you. The SWB webpage provides the means to request assistance.
  • SWB on Facebook
  • SWB on Twitter
  • SWB on LinkedIn

The American Evaluation Association is celebrating Statistics Without Borders Week. The contributions all week come from SWB members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

· ·

My name is Staci Wendt and I am a Research Associate at RMC Research in Portland, Oregon. Last year, I completed my Ph.D. in Applied Psychology at Portland State University. After finishing my degree, I was concerned about how to stay current with statistical literature and how to practice techniques that I learned in school, but wasn’t currently using in my work.

Hot Tip – One day, a friend was talking with me about her fiction book club and I had an “Aha!” moment—a book club where we discussed statistics!

Who: We have a small group of people with varying knowledge and experience related to statistics and research methods. Our group is comprised of 6 members, which eases scheduling and allows each of us the opportunity to meaningfully contribute.

When: While our regular meetings are held monthly, we are also available to each other via email throughout the month. The email discussions allow for quick feedback on questions or issues that might arise within our day-to-day work.

What: At our first meeting, we discussed our goals and expectations for the group, brainstormed a list of topics we wanted to discuss, and decided on the format for our group. After this discussion the group decided that in order to make the group both useful and doable we would meet monthly but vary the meeting type. On odd-numbered months, we have formal meetings, where we discuss a pre-determined topic (such as Structural Equation Modeling). We take turns facilitating these formal meetings. The facilitator is responsible for selecting pertinent sub-topics of the theme (e.g., model fit, assumptions of the statistical test, how-to) and assigning them to each member. Each member is then responsible for creating a small “cheat-sheet” on that topic and presenting the information at our meeting. Our presentations are mostly casual in order to encourage a good environment for discussion. We also try to bring pertinent “real-world” examples, either from the literature, or from our own work. On the even-numbered months, we have informal meetings. At these meetings, we bring any specific question or topic that we want to discuss, or review information from the previous meeting. The main difference between the formal and informal meetings is that we don’t have any preparation work for the informal meetings.

Where: We rotate meeting at different group members’ homes for the formal meetings. This allows one person to take notes (which are later distributed to the group) and we have room for reference books. For the informal meetings, we try to meet at restaurants, to add to the relaxed nature of the meeting.

The most important thing is to set group goals, make adjustments as you try it out, and HAVE FUN!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

My name is Allen Blair and I’m not an evaluator per se, but rather a statistician. I work with evaluators to assist with statistical analyses and I am posting to aea365 to share three favorite blogs for those who are ‘numbers people’, although I think that they are actually more useful for those who find it difficult to think in terms of numbers. Each of the following interprets our everyday lives through numbers. I’m going to take the same approach that Alex da Silva did back in May when recommending sites for expanding your capacity with excel – beginner, intermediate, and advanced, with a couple of examples from each:

Beginner Rad Resource – The Numbers Guy at the Wall Street Journal: Carl Bialik is the numbers guy. His weekly Wall Street Journal column “tells the story behind the stats.” Bialik holds a mathematics degree from Yale and his everyman explorations, posted approximately weekly, are based in sound mathematics.

Recent Example Posts:

  • Mind the Median
  • Sexual Stats in the Post-Kinsey Age
  • NCAA Brackets Math

Intermediate Rad Resource – Three-Toed Sloth: I’m baffled by the name, but the content is great. Cosma Shalizi, an assistant stats professor at Carnegie Mellon, posts a couple of times a month with a mix of commentary and exploration of issues in statistics. All of it comes with a touch of academic wit.

Recent Example Posts:

  • Knights, Muddy Boots, and Contagion; or, Social Influence Gets Medieval
  • Of the identification of Parameters
  • Your City’s a Sucker, My City’s a Creep

Advanced Rad Resource – Social Science Statistics Blog: “This blog makes public the hallway conversations aboutsocial science statistical methods and analysis from the Institute for Quantitative Social Science and related research groups” at Harvard University. The content can be all over the place, but it offers great resources usually in short casually-written pieces.

Recent Example Posts:

  • A search engine for figures
  • Can a single case be used to test theory?
  • A Cure for the Regex Headache

Share your favorite stats blog via the comments!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Older posts >>

Archives

To top