Hello, AEA365 community! Liz DiLuzio here, Lead Curator of the blog. This week is Individuals Week, which means we take a break from our themed weeks and spotlight the Hot Tips, Cool Tricks, Rad Resources and Lessons Learned from any evaluator interested in sharing. Would you like to contribute to future individuals weeks? Email me at AEA365@eval.org with an idea or a draft and we will make it happen.
Sometimes it is helpful to be very precise. On the other hand, false precision – giving the air of precision without the data to back it up – can be irrelevant and/or misleading at best and, at worst, can destroy, rather than enhance, the credibility of your evaluation – and of you. Hi, I’m Burt Perrin, and I’d like to discuss what considerations such as these mean for evaluation practice.
If one is undergoing brain surgery, one would hope that this would be done with precision based upon established knowledge about the procedure. That said, a surgeon can be no more precise than the underlying data about their procedure permits. Similarly, an evaluator can be no more precise with their conclusions than their underlying data permit, and yet attempting this is where too many evaluations go wrong.
A recent study drawing sweeping conclusions about the practices of evaluation managers is a case in point. The study was based upon a response rate of less than 2%, with the researchers saying that the small sample size did not matter as this was “an exploratory study.” There were numerous other flaws such as questionable interpretations of responses to many of the questions. Despite this and numerous other shortcomings, findings were reported using multiple decimal points implying a level of precision that was unjustified based on the research methods. Such “precision” was hardly justified by the underlying data. This, unfortunately, is not an isolated example.
Too often, analyses are presented to two decimal places because this is the SPSS default, which, unless the underlying data set permits this degree of specificity, would be meaningless and misleading. Surely, with, for example, a typical margin of error of at least 3%, the absurdity of such spurious precision should be evident.
There are dozens of other factors that similarly might limit the ability to be overly precise, such as sampling error, measurement and instrument error, considerations regarding questionnaire design and manner of administration that have been shown to affect responses. These include, but go beyond, question wording, question ordering, variables concerning questionnaire implementation in practice, coding errors, variable (mis)understanding of the questions and response options, demand characteristics of the survey context, timing, and many more. With less than a 100% response rate, there is a very strong potential of non-response bias distorting findings. (I view this consideration as particularly important, and I will return to this in a separate blog posting. Watch out for it.)
Meaningful analysis needs to consider possible influencing factors such as the above, bearing in mind that even the most accurate measurement can only be specified within a range. When factors such as these are not taken into consideration, the meaningfulness of the resulting data, analyses, and conclusions can well be called into question, irrespective of the number of decimal points used and reported significant levels.
Effects of misplaced precision can distort overall approaches to evaluation, and to program design and implementation, leading to a focus on what is easy to measure rather than what is most important.
A key learning: Generally, the more complex an issue, the more difficult it is to specify with precision, with danger of falling into what Scott Chaplowe refers to as the accountability trap. <https://socialinnovationsjournal.com/index.php/sij/article/view/704>
I’ve spoken and written about this topic extensively, including in the AJE (https://journals.sagepub.com/doi/abs/10.1177/109821409801900308 _. As John Maynard Keynes has observed: “Better to be approximately right than exactly wrong.”
Burt Perrin (Burt@BurtPerrin.com), formerly from Canada and currently living in France, has been engaged in evaluation internationally for many years, for example as a leader and founding member of the CES, EES, and AEA. He has authored another AEA365 blog posts,
. including about the backfire effect (Facts and evidence make no difference – or worse) and about bureaucracy and evaluation (Can evaluation help make bureaucracy more responsive – or is it part of the problem?).
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.