Hello AEA community, my name is Patricia Campion, PhD in Sociology and currently an independent evaluator with twenty years of experience. A valuable lesson I have learned through the years is that we shouldn’t be afraid to keep things simple to be mindful to our audience, and that it doesn’t mean we have to dumb things down.
While we all like to showcase our expertise with advanced methodologies, since it is so central to our profession, many in our audience are intimidated or confused by methods jargon. To reach them efficiently, however, we do not need to forgo methodological rigor.
I learned to combine both complex and simple reporting techniques while evaluating an Intimate Partner Violence (IPV) prevention program for Sunrise of Pasco County (https://www.sunrisepasco.org/), in Central Florida. The program was funded by a DELTA FOCUS grant with CDC (https://www.cdc.gov/violenceprevention/deltafocus/index.html) over five years.
The evaluation relied on a pre-posttest for students enrolled in an after-school program. The logical approach to analyze this type of data was a nonparametric means test. For Sunrise’s annual report, I carefully outlined the logic behind the test I had chosen to use, the Wilcoxon Signed-Ranks test (https://www.youtube.com/watch?v=dkobjvhxTro). In case anyone who read the report knew about the more commonly-used t-test, I also included it and explained why it wasn’t the best choice.
The Wilcoxon test showed significant improvement on some of the measures. The program staff and state evaluators, however, weren’t as excited as I had expected them to be: They did not connect with my SPSS result tables and its many statistics.
A conversation with the program coordinator led us to consider a simpler approach. The statistical tests were deemed necessary, but we needed to add a more impactful visual tool. So, I went back to basics, i.e., the data at the heart of the tests. Since we were looking at differences in average results on a number of survey items, I used Excel to build a simple graph showing the average class scores for each item in the pre- and posttests. The pretest showed as a red line, the posttest as a blue line. Each time the blue line was higher than the red line, the program had made a difference.
The simpler display, of course, did not have the statistical rigor of my SPSS tables. Not every difference it showed was real in a statistical sense. However, everyone connected to it. And they were not interested in looking at each individual item on its own. They looked at the graph in a holistic fashion, as showing that the program was able to make a difference, though not always and not to the same extent for all topics.
- Be ready for people at various levels of quantitative literacy to read your reports: Mix statistical and visual content at different levels of expertise.
- Don’t reinvent the wheel. If you already use Excel, it provides a reliable source of quality graphs. It allows you to easily import data from SPSS and copy/paste graphs into other Microsoft apps.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.