Hi! I’m Lyn Boswell, a member of the Program Design Topical Group. I have worked abroad in monitoring and evaluation (M & E) in various cultures.
In the U.S., designing metrics to determine effectiveness of programs requires avoiding different “bias minefields” in achieve objectivity and validity. For example, the controversial topic of “defunding the police” is laden with much bias. As a result, it is impossible to create program design with any objective metrics.
Having the goal to “defund the police” is a political posture. It does not result in realistic measurements except for reducing budget and personnel. Once that’s done, the goal is considered achieved. But, is that really the goal? Absolutely not. The question is about whether the program, defunding in this case achieves the intended goal of a problem to which we would like a resolution. I say this because we all know that defunding is not a solution for most law enforcement.
Thus, “defunding the police,” which is heavily laden with bias would be less biased with clear goals such as:
- less lethal methods used for unarmed individuals;
- less lethal methods towards armed or unarmed mentally ill; and
- more de-escalation training.
Notice, that race is not tied with any of these goals. However, skin color has often been the “motive” for defunding the police movements. From an evaluation perspective, it is not a valid measurable goal.
Hot Tips:
- As a professional program designer and evaluator, the most important information needed is discerning what is the “real goal” you are trying to measure?
- You may need to peel back multiple layers of bias to understand what goal needs to be achieved (i.e., individual, organization, or community). Then, you can talk about solutions that are achievable/measurable.
Consider your own personal biases an evaluator. You may find the results of the Harvard Implicit Bias Test interesting: https://implicit.harvard.edu/implicit/takeatest.html
How have your biases impacted your own interpretations of project goals and outcomes?
The American Evaluation Association is celebrating Program Design TIG Week with our colleagues in the Program Design Topical Interest Group. The contributions all this week to aea365 come from our Program Design TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hi Lyn,
Thanks for this post. It perplexes me, so I feel compelled to ask you a few clarifying questions as I seek to understand your logic.
1. Is evaluating a program or policy related to the ‘defund the police’ movement any more bias-laden than any other evaluation? The UN’s Sustainable Development Goals (SDGs), or the ‘No Child Left Behind’ (NCLB) policy, for example, seem just as vague, controversial, and slogan-inflected. Do you find it equally impossible to create objective measures of those evaluands? If so, how and why can one ever evaluate anything?
2. Is the defund the police movement any less evidence-informed than other programs or policies? Are you familiar with the specific evidence base and historical precedent for this policy proposal, dating back at least to DuBois (1935; see also Vitale, 2017)? If not, why did you choose it as an example? I think that if you took a few minutes to look into the defund the police movement, you would learn that there are very specific and measurable objectives as part of the initiative, though (like with all social change processes) some level of disagreement as to methods and objectives remains. In the absence of those few minutes of research on your part, you seem to be critiquing a straw person version of the movement, rather than the movement itself.
3. Are supporters of the defund the police movement inherently more likely than most to be mindlessly parroting slogans without applying critical evaluative thinking to the evidence? Is it plausible that the thousands of policymakers endlessly repeating claims to achieve the SDGs or NCLB goals might be less-well-informed about the rationale and evidence base for their goals, with all their concomitant paradigmatic assumptions? I think it is possible or even likely that evaluators, researchers, and program implementers advocating for programs and policies related to the defund the police movement are actually well-informed critical thinkers; your implications to the contrary thus risk patronizing and gaslighting them.
4. When you write, “… we all know that defunding is not a solution for most law enforcement,” doesn’t the epistemic and rhetorical move you pull with that ‘we all know,’ absent any empirical evidence, risk becoming an example of exactly the type of bias you purport to warn against? Who is the ‘we’ in that sentence? How, precisely, do ‘we all’ know that? Show me the evidence, or I fear you might be relying on your own unexamined bias–rather than empirical evidence and theoretical framing–to support your claim.
5. In a later comment, you imply that the defund the police movement is unpopular; there is evidence to suggest that is true, but I fail to see how that has any bearing on the degree of bias risk in evaluating programs and policies related to the movement. The SDGs are unpopular in many places, and NCLB was extremely unpopular in many schools, yet that did not make them any less evaluable. Also, the abolition of slavery was unpopular in the Southern U.S., yet that did not make it an undesirable ethical proposition to pursue.
6. What do you mean in the past paragraph about race? I tried to follow your argument but got lost. Please help me understand your point about race and its relation to defunding the police.
Thanks!
Tom
What an “interesting” choice to place quotation marks around the words defund the police. Placing quotation marks around a set of words can be used to demonstrate that they are someone else’s words; yet you do not quote a specific person or organization, so I don’t believe that was your intended usage. The other usage for quotation marks is to connote irony, sarcasm, or emphasis (aka, scare quotes). Writers frequently use scare quotes (whether intentional or not) to demonstrate that something is silly, or to imply skepticism or disagreement (https://en.wikipedia.org/wiki/Scare_quotes). As soon as you placed quotations around the words defund the police, I knew you disagreed with the abolition movement. How ironic that you selected the use scare quotes in a piece attempting to critique the bias behind the goal of defunding the police.
In fact, the goal to defund the police is very real and literal. The abolition movement wishes to decrease the amount of funds that go to police departments and divert those funds to other social services. Despite what you state in your blog post, defund the police is actually an example of a very concrete and measurable goal.
It would be irresponsible as an evaluator to suggest different goals to our clients and colleagues just because we do not agree with their original goals. Your suggestions of less lethal methods and de-escalation trainings are simply different than defunding the police. Those are goals of reform movements, but not abolition. As I mentioned above, defunding the police is not a euphemism, it’s a literal goal.
Just because you do not agree with something does not mean it is not real. Similarly, just because you do not agree with a goal, does not mean that it is a poorly written goal.
For future readers of this blog post, here are some observations I want to leave for posterity.
The author of this post has made a point of raising the issue of bias, describing those who support defunding police (otherwise know as the abolition movement) as having a political posture, which she says results in a biased position that defies measurement. What she does not acknowledge, including in the comments posted here to date, is that her own apparent position of continuing to fund law enforcement is equally a political position (describing in a comment that this movement is one which “vilifies” law enforcement). She is applying a double standard by suggesting that the status quo is somehow more neutral, apolitical, and objective than any position which is counter to it. She has, in fact, in making this post gone out of her way to make a political point, while framing it as a “thought exercise” despite it being a serious life and death issue.
She also makes unsubstantiated claims about the goals and the viability of the abolition movement and is demonstrably uninformed about the history of this movement and the considerable scholarly work that has already been done and continues to be done on how abolition might be achieved and what its impacts can be. She has no credentials or experience to speak of that she has offered or that are available online that would qualify her to make the statements she does. Other experts who are better consulted in this area include Mariame Kaba and Ruth Wilson Gilmore, for whom this is an area of significant scholarship and practice.
Her statements about what make something “a valid measurable goal” have been contested by a number of experienced evaluators (including some who have commented here), but if you are reading this you are likely someone with an interest in evaluation who is more than capable of assessing for yourself whether her statements sound credible or accurate to you.
The inclusion of the Harvard IAT measures is confusing and seems completely unrelated to the subject of the post beyond the fact that both refer to bias. Cognitive association in timed response tasks is a different form of bias entirely.
The context of this post is also that at the present time there are a number of people who are concerned about the “politicization” of evaluation and the erasure of more “moderate” or “conservative” viewpoints. The author of this post may not feel that way (I have no insight on that), but those who do will certainly interpret these critiques of her post (despite them being based again on sincere criticisms of her definitions, assumptions, and relevant expertise) as unwarranted attacks.
Hmmm…looks like there might be some confusion here. “Defund the police” is not a program? And yet, centuries worth of theory underlie it as a strategy for reducing violence in general–particularly violence against marginalized groups, and especially against those racialized as Black.
Let’s hope that Topical INTEREST Groups take greater responsibility for the quality of work that they include in their weeks before such impoverished perspectives go public again. The wait list is far too long for AEA365 for days to be wasted on entries that are hastily written, with few citations, no due diligence, and even no copy-editing.
This is a disappointing post right after a week focusing BIPOC women. This post completely missing the point of call to defund the police. The goal of defunding the police is not simply to reduce budget and personnel, but to use budget taken from the police and invest it in upstream community supports that prevent social ills and lift up oppressed groups. The goal of defunding the police is to reduce the disproportionate number of death by police that happen to Black and other minority groups. That is certainly valid and measurable.
Hi Paul,
I believe you have clearly made my point, as you have stated what you believe the goals to be, and perhaps that is correct in some communities. It is not, however, the goal for other communities. If you have had the opportunity to read other comments here, you will see that the goal is no police, no jails, no prosecution of vice crimes (sex trades, drug trades, etc.). That is some of what defunding means to them and those communities.
We go back to the issue of (which seems to be consistently missed) approaching goals with less bias. Recognize that not all communities want anything to do with defund the police. That needs to ok. To the chagrin of law enforcement, I mentioned in one of my replies that “back the blue” was also bias-laden. Law enforcement disagrees but not all. I see that “back the blue” presumes the bias that the blue can do no wrong, and that we support their actions; and defunding is not different in that regard. “Back the Blue” also evokes an “us versus them” ideology. It’s fine to support law enforcement and at the same time, completely condemn bad policing or bad laws/methods, and I’m back to “What goal(s) is this (back the blue) aimed at that is achievable and measurable?” How do we reduce bias in those goals? Again, you will have different goals and interpretations. Given our roles, why can’t we assist in clarifying the activities and goals? And yes, again, I was told what the goals were for “Back the Blue,” in the same way you stated what “Defund” goals were, and trust that I did not receive the same nor consistent answer.
As demonstrated by our colleagues, these articles links below explain program design further in defining programs, definitions, aims (which I call goals), approaches, vision, team-building, engagement, reasoning, collaboration, and “how this is supposed to work (Shaffer blog below).”
https://aea365.org/blog/program-design-tig-week-the-power-of-document-models-in-program-design-by-jacqueline-singh/
https://aea365.org/blog/program-design-tig-week-engaging-program-stakeholders-in-theory-of-change-and-logic-modeling-by-patricia-moore-shaffer/
https://aea365.org/blog/program-design-tig-week-what-is-a-program-by-catherine-nameth-jacqueline-singh/
Because there are definitional goal issues, and because Defund the Police has such varying “goals” and an auto-bias in the movement, my challenge is to recognize bias, and aim to reduce our own bias when we are called to be professional evaluators.
I liked Asma’s words “culturally responsive.” https://aea365.org/blog/program-design-tig-week-evaluations-for-program-development-lessons-learned-from-six-months-of-virtual-community-building-by-asma-m-ali/ Though she speaks specifically to virtual environments, the foundational objectives of an evaluator still does not change.
Jacqueline’s hot tips referenced in her blog, and the 4 bullet points of the Catherine/Jaqueline blog go back to fundamentals. Am I suggesting the fundamentals be changed? Absolutely not. Am I suggesting that we ignore our understanding on theories of change, or our pillars of logic models? Absolutely not. What I am suggesting is that we need to be careful and critical (watchful) of the biases that surround us so as not to be duly influenced by the same biases that could/would create goals/activities/programs that already measure a design (question or activity) that is inherently biased. Even the sample goals I provided have presumptions and a bias (fine if that is what the community wishes, yes?) especially when I mentioned less lethal methods. Since there is a word limit on the blog (but not on comments), you didn’t get the opportunity to read that. I wondered if you had the opportunity to read my response to David’s comment in here.
My initial blog topic may be more suited to a paper or presentation where much of this would be fleshed out. Within my professional capacity, my first reaction is to always check the bias at the door in order to ensure that the questions, measures, activities, etc. do not inherently provide a biased program design.
Paul, I do thank you for taking the time to critically read this blog and making your comment to me.
Kind regards,
~Lyn
Perhaps the author of this blog left out some crucial background, or was in a hurry??
Hello Mr David,
Not in a hurry, no. The initial draft was almost 600 words. Perhaps better suited in a full medium of a paper or presentation. At the beginning of my intro, I did mention the international scene. I have learned that what one hears and sees in media may not be the whole picture, and that our own biases tend to influence how we interpret the “message.”
The point was not to offer solutions and continue a movement that vilifies law enforcement through this “defund the police,” but recognizes the fact that not all towns, cities, and citizens believe in this or want this for their own communities. As communities work through what “defund the police” means to them, as impartial evaluators, we need to get to how we can assist in determining the goal and work backwards with less emotion, and more objectivity.
The connotation of “defund the police” groups the good and bad, the successes and the failures, and presumes that it is the police at fault. I don’t know that the police are at fault, I don’t know if the community is at fault, I don’t know if education, training, mental health evals, rule of law, courts, faith, human nature, etc. are actually the heart of the issue.
All I am presenting are potential samples, without knowing what the real goals are, and asking us to come into hot button topics with less bias, and not be caught up in what others want us to see, but to understand and evaluate the issues on its own merit. This is also a very complex and complicated statement which means different things to different communities. We have to be brave enough and honest about how a simple statement “defund the police” may not be as it seems.
I sincerely appreciate you reading and commenting.
I wish that you had managed a way to say this in your blog exactly as you have replied to my brief comment. I agree with your observations, and wish you well in your future writing. David Robinson
I don’t understand why the goal of ‘defund the police’ is seen to be laden with much bias. Like any goal it needs to be unpacked for an evaluation. Doing so mght well articulate more specific intended outcomes and processes that are encapsulated in this brief term.
I also disagree with the assumption that evaluation is necessarily about developing measures for goals, at least in the usual understanding of what a measure is. I see my role in evaluation as helping people to bring together relevant, valid evidence and explicit, defensible values to form evaluative judgements about what success looks like, and whether this has been achieved. A lot of my efforts go into persuading people not to dumb down their goals into something that can be readily measureable, but definining goals (and principles) clearly – and then working out how to bring evidence to bear on it. In many cases a well-constructed rubric that can synthesise diverse and changing evidence (see https://www.betterevaluation.org/en/evaluation-options/rubrics) is more valid and useful than a few metrics which risk common negative results from performance measurement which include mmis-representation, goal displacement, data corruption, mis-direction of effort and over-correction (see https://www.betterevaluation.org/en/resources/ANZSOG/risks_of_getting_performance_measurement_wrong).
Hi Patricia,
I believe we are actually saying the same things but in different ways because I do agree with much of your comments.
The reason I call this topic bias laden is for the very reason that it does need to be unpacked. “Defund” has a negative connotation already. “Police” identifies with the 800,000+ law enforcement personnel as the target. It emotes strong feelings either way…and I think if that is the case, then the statement is truly biased. No worries as I would easily proffer “Back the Blue” as bias laden as well. I just didn’t choose that for this particular blog.
I completely agree that we can’t dumb down goals for easy measurability. I offered samples that could be goals. The issue is both complicated and complex which is why we, as professional evaluators, will have the same foundational logic models. I think we work the models differently and still end up with the same or similar diagram.
I am usually brought in after the goals and activities have been established with no baseline, no evidence, no community input, etc. At that point, we usually have a difficult conversation about assumptions, validity, and a re-assessment of the activities to achieve the goals. Some communities have already reacted to the “defund the police” movement without consideration of the unintended consequences. Without a well thought out program design in these instances, this just becomes a knee-jerk reaction instead of a carefully thought out establishment of realistic goals (the unpacking, as you mentioned) and the likelihood of failure to achieve the goals.
Even from your examples (websites), you would agree that the models will look different in any of the communities who want to “defund.” (This means different things to different communities.) Then there are those exemplary communities who do not want to “defund” because they have already implemented successful policing and do not understand the movement at all (perhaps then eliciting “Back the Blue”?).
Regardless of our own “feelings” about the topic, I find it critical to reduce our own biases toward a topic as impartial evaluators or impartial change agents in assisting communities to develop achievable realistic goals, and good performance measures.
For these and other reasons, I do not necessarily disagree with your comment and I appreciate your added insights and critique.
Abolition is an evidence-based movement. And here are some clear goals explicitly outlined by the abolition movement that unpack what “defund the police” means, in ways that are very clearly implementable with outcomes that could be tracked and assessed both quantitatively and qualitatively: https://static1.squarespace.com/static/5edbf321b6026b073fef97d4/t/5ee0817c955eaa484011b8fe/1591771519433/8toAbolition_V2.pdf
Thanks for this post. Many of us simply repeat political mantras without considering their real-world applications or how they can have different meanings for different people. Applying your perspective to these issues can do much to identify and reduce unintended consequences of a program or policy.
Hi Colin,
Thank you for interpreting this the way I had intended to convey the message.