Evidence of What, for Whom? The Socially Contested Role of Algorithmic Bias in a Predictive Policing Tool
Authors:
Marta Ziosi,
Dasha Pruss
Abstract:
This paper presents a critical, qualitative study of the social role of algorithmic bias in the context of the Chicago crime prediction algorithm, a predictive policing tool that forecasts when and where in the city crime is most likely to occur. Through interviews with 18 Chicago-area community organizations, academic researchers, and public sector actors, we show that stakeholders from different…
▽ More
This paper presents a critical, qualitative study of the social role of algorithmic bias in the context of the Chicago crime prediction algorithm, a predictive policing tool that forecasts when and where in the city crime is most likely to occur. Through interviews with 18 Chicago-area community organizations, academic researchers, and public sector actors, we show that stakeholders from different groups articulate diverse problem diagnoses of the tool's algorithmic bias, strategically using it as evidence to advance criminal justice interventions that align with stakeholders' positionality and political ends. Drawing inspiration from Catherine D'Ignazio's taxonomy of "refusing and using" data, we find that stakeholders use evidence of algorithmic bias to reform the policies around police patrol allocation; reject algorithm-based policing interventions; reframe crime as a structural rather than interpersonal problem; reveal data on authority figures in an effort to subvert their power; repair and heal families and communities; and, in the case of more powerful actors, to reaffirm their own authority or existing power structures. We identify the implicit assumptions and scope of these varied uses of algorithmic bias as evidence, showing that they require different (and sometimes conflicting) values about policing and AI. This divergence reflects long-standing tensions in the criminal justice reform landscape between the values of liberation and healing often centered by system-impacted communities and the values of surveillance and deterrence often instantiated in data-driven reform measures. We advocate for centering the interests and experiential knowledge of communities impacted by incarceration to ensure that evidence of algorithmic bias can serve as a device to challenge the status quo.
△ Less
Submitted 13 May, 2024;
originally announced May 2024.
Ghosting the Machine: Judicial Resistance to a Recidivism Risk Assessment Instrument
Authors:
Dasha Pruss
Abstract:
Recidivism risk assessment instruments are presented as an 'evidence-based' strategy for criminal justice reform - a way of increasing consistency in sentencing, replacing cash bail, and reducing mass incarceration. In practice, however, AI-centric reforms can simply add another layer to the sluggish, labyrinthine machinery of bureaucratic systems and are met with internal resistance. Through a co…
▽ More
Recidivism risk assessment instruments are presented as an 'evidence-based' strategy for criminal justice reform - a way of increasing consistency in sentencing, replacing cash bail, and reducing mass incarceration. In practice, however, AI-centric reforms can simply add another layer to the sluggish, labyrinthine machinery of bureaucratic systems and are met with internal resistance. Through a community-informed interview-based study of 23 criminal judges and other criminal legal bureaucrats in Pennsylvania, I find that judges overwhelmingly ignore a recently-implemented sentence risk assessment instrument, which they disparage as "useless," "worthless," "boring," "a waste of time," "a non-thing," and simply "not helpful." I argue that this algorithm aversion cannot be accounted for by individuals' distrust of the tools or automation anxieties, per the explanations given by existing scholarship. Rather, the instrument's non-use is the result of an interplay between three organizational factors: county-level norms about pre-sentence investigation reports; alterations made to the instrument by the Pennsylvania Sentencing Commission in response to years of public and internal resistance; and problems with how information is disseminated to judges. These findings shed new light on the important role of organizational influences on professional resistance to algorithms, which helps explain why algorithm-centric reforms can fail to have their desired effect. This study also contributes to an empirically-informed argument against the use of risk assessment instruments: they are resource-intensive and have not demonstrated positive on-the-ground impacts.
△ Less
Submitted 10 June, 2023;
originally announced June 2023.
