Can machine learning algorithms inherit human racism? In modern empirical research, shifting from explanatory modeling to pure predictive machine learning can allow researchers to unlock incredible social welfare gains, eliminate personal bias and optimize institutional decision making. However, if predictive algorithms operate in a theoretical vacuum, detached from an understanding of how data is generated, they risk automating systemic bias. The most valuable empirical research demonstrates that the intersection of theoretical explanations and predictive machine learning delivers the most accurate answers about the world.

Pretrial bail decisions offer a clear illustration of a machine learning algorithm outperforming human judgement. In the U.S. criminal justice system, judges reliant on their subjective intuition evaluate whether arrested individuals are able to await trial at home or whether they flee and commit a new crime. Kleinberg, Lakkaraju, Leskovec, Ludwig, and Mullainathan (2017) in their foundational study Human Decisions and Machine Predictions demonstrated that human judges majorly underperform in evaluating this question. Not only do they underperform, they systematically mis-rank the defendant risk. The authors trained a machine learning algorithm using gradient boosted decision trees on over 750,000 arrest records from New York City from the years 2008 to 2013 and replicated them on a national dataset of over 150,000 felony cases. Assessing the accuracy of the judicial decisions, the authors found that “[u]sing just data available to the judges at the time of the pretrial hearing, those in the top 1% of the predicted risk distribution have an observed crime rate of 76.6%, and yet are released by judges at a rate of 52.5%” (Kleinberg et al., 2017). Furthermore, the findings show that if we replaced judicial decisions with algorithmic prediction “crime can be reduced by up to 24.8% with no change in jailing rates, or jail populations can be reduced by 42.0% with no increase in crime rates” (Kleinberg et al., 2017). The benefits of machine learning models are undeniable. When algorithms are applied to concrete prediction policy problems, they optimize institutional decision-making in ways that human intuition alone cannot match.

The success of predictive applications in pretrial decisions, however, led to a dangerous assumption: that machine learning algorithms could completely replace explanatory models altogether in solving prediction policy problems. Commercial health systems in the U.S. use a predictive machine learning model to help manage patient care and to enroll high risk patients with complex needs in “high-touch care management programs”. In their paper Dissecting racial bias in an algorithm used to manage the health of populations (2019), Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan critically examine a prediction algorithm applied to roughly 200 million people in the U.S. annually. The algorithm decides who gets access to specialised healthcare based on a ‘health risk score’. “Patients above the 97th percentile are automatically identified for enrollment in the program” (Obermeyer et al., 2019). The authors looked at algorithmic risk scores of 50,000 patients and linked those to their actual health records and insurance claims. The results showed that Black patients who were automatically enrolled at the 97th percentile had 26.3% more active chronic conditions, diabetes and hypertensions. The algorithm displayed a severe systemic racial bias. If the algorithm was to be corrected so that equal health needs would receive equal risk scores, the percentage of Black patients automatically identified would more than double, from 17.7% to 46.5%.

The issue stems from a fundamental flaw in the problem formulation: the developers trained the algorithm to predict future healthcare costs as a proxy for health needs. In the real world, healthcare spending does not equal sickness. At any given level of health, Black patients, due to structural inequalities, “generate $1,801 less per year than White patients” (Obermeyer et al., 2019). As the machine learning model treated ‘cost’ as equal to ‘sickness’ Black patients had to be much sicker to be admitted to the program, than White patients. Crucially, the machine learning model never used race as an input feature. Operating in a theoretical vacuum, the algorithm misinterpreted historical spending disparities as true health differences, automating systemic racial bias. Choosing a convenient, easily measured data proxy, such as spending, rather than a direct measure of illness, creates biased predictions that fail to reflect true health needs.

Following the study, authors collaborated with the algorithm manufacturer to fix the bias. They altered the training model by switching the target variable from pure cost to an index combining health and cost predictions. This reduced the racial health disparity conditional on risk score by 84%. This fix highlights an essential parallel with Kleinberg et al.’s model. A central concern for the authors was ensuring that algorithms did not aggravate racial disparities. However, they proved that algorithms can be explicitly trained to enforce racial equity. In their model, the algorithm was constrained to match or reduce the minority share in jail, while still achieving a 22.7% to 24.6% reduction in crime.

All this goes to show that predictive tools can serve as not only a powerful driver of institutional efficiency but also a force for racial fairness. Predictive technology outperforms human intuition, cutting crime or jail population while reducing racial disparities in pretrial detention. However, this is only realized when algorithms are guided by explanatory theory and structural context. Without theoretical oversight, machine learning models risk predicting flawed proxy variables, such as healthcare cost instead of true sickness, and automate real world discrimination. Ultimately, prediction and explanation go hand in hand: theoretical explanation serves as a guiding tool which steers predictive models in the right direction to deliver the fairest outcome.

Written by CERSP intern  Amélie Benková.