The Devil You Don't Know
The Devil You Don't Know

The Devil You Don't Know

A data scientist uncovers the power of hidden algorithms that carry out digital injustice.

June 8 th 2017

A pit bull lunging to the full extent of its chain in its attempts to rip you apart is disturbing in its own, direct way. And hazards that we barely see can be even more disturbing—think of Steven Spielberg's use of a barely seen menace in Jaws, or Francis Lawrence's use of mannequins in I Am Legend. Just a hint or a glimpse of the shark, or of the mutated humans, is enough to terrorize us.

But completely unknown hazards, even if they are horribly dangerous, don't frighten us at all. What you don't know can't scare you. Sometimes this can work in your favour—think of Shasta walking on a foggy path crossing the pass into Archenland without seeing the sheer cliff—but more often not knowing the danger can lead to disastrous consequences. You want someone—or some sign—to point out the dangers, so you can adjust your behaviour properly. That's why we have developed labels for poison, warning signs for radiation, and laws for financial management; they protect us from dangerous things that look harmless.

What you don’t know can’t scare you.

Data scientist Cathy O'Neil uncovers how the unseen presence of mathematical instructions that process our data secretly harms millions of people. These algorithms, says O'Neil, are no less dangerous because we don't see them. In fact, they are "weapons of math destruction" that we don't even know exist until they explode. Her description turns our blissful ignorance of mathematical algorithms into motivating fear, maybe even anger.

Since most of us remember mathematics as harmful only to our high school grade-point averages (or our sanity), it may seem like overblown authorial enthusiasm to claim that massive destruction is arising from equations. Is this the equivalent of a book warning about the addictive power of knitting?

No. O'Neil is deadly serious.

We have tied computational systems to nearly every aspect of our lives. Some of these are good. Airlines use mathematical systems to improve travel efficiency and safety; banks use financial algorithms to enable us to access money around the globe; researchers also use algorithms to sort through mounds of data to identify patterns in health, or traffic flow, or to detect fraud. Algorithms can be good.

Once a model is made, then it becomes truth and the conclusions from it are infallible

But these same systems, built around sets of mathematical instructions that make decisions about incoming data, also steer resources away from people who need it, unfairly change credit scores, identify teachers to be fired without regard for context or external variables, increase insurance rates without due cause, or unjustly consign prisoners to extended jail terms. All of these things happen because of risk-evaluation tests that are "built in" to an algorithm that banks, courts, school boards, and others rely on to make decisions. The problem is, these same institutions aren't actually aware of the assumptions and calculations that lead the program to recommend the decision. In some cases, they aren't even allowed to see the calculations.

Mathematical models that interpret computer data and then issue instructions can help us make sense of very complex processes and phenomena. A lending agency, for instance, needs to evaluate how risky it will be to make a loan to a young family looking to buy a home. In order to enhance the quality of the decision, past history, address, income, assets, and other data is entered in to a computer model that produces a rating or interest rate that maximizes profit for the bank. In this oversimplified scenario, neither the bank worker entering data nor the applicant has control over the instructions coded into the process—they just put information in and get a result out. The problem is, those instructions may be far from fair in a given case.

Or take another example: the use of algorithms to sort résumés. When the algorithm used for thousands or millions of job applications that are sorted by computers has a coding bias against race or income built into it, significant harm can result. The challenge facing the black person or the poor person who is overlooked for a job without recourse, or even knowledge of why they were overlooked, is that the program is given unique status—above humans—as a decision maker.

Evaluating the Evaluators

The highly respected statistician Leo Breiman has commented on this faith-like trust in machines and data. He notes that in prior decades "the belief in the infallibility of data models was almost religious. It is a strange phenomenon—once a model is made, then it becomes truth and the conclusions from it are infallible."

When we build models, we bake in assumptions, sometimes without knowing it. What we need now are improvements not only to our programs but also to our ability to evaluate the merits of a given level of trust we place in our algorithms. This is particularly true, as O'Neil notes, if the programs are woven into large systems that affect millions of people. One way to query the level of blind faith we put in a program is to consider the extent to which the software provides decision support to, say, a bank employee, compared with the extent to which the algorithm makes the decision with no recourse to the decision-making powers of that same bank employee.

When we trust built technologies like bridges and petroleum pipelines, the equations and decisions that constituted their design and engineering are hidden from us, but their functions (or dysfunctions) are much more visible to us—bridges hold up or collapse, pipelines retain or release their contents, and so on. But hidden algorithms in banking, insurance, prisoner risk evaluations, and elsewhere are considerably less transparent in their effect. Because of this, O'Neil argues, their failures may go undetected and will wreak destruction long before anyone notices, much in the way DDT did before Rachel Carson traced the nefarious patterns of pesticides in Silent Spring.

We've been taught to worry about robots in the wrong way. In Terminator 3: The Rise of the Machines we are asked to imagine humanity doing battle with armed robots. But this mechanical apocalyptic vision is a distraction from the real work of data machines that don't need sentience to do harm. They carry neither guns nor swords but live among the unseen patterns of logic encoded in server farms and corporate data rooms, informing decisions and exerting far greater powers.

Their failures may go undetected and will wreak destruction long before anyone notices

O'Neil identifies weapons of math destruction (WMDs) as having three characteristics: Opacity, scale, and damage. From an interrogative angle, these three characteristics can be framed as questions: Are those affected by the algorithm aware that they are being evaluated by it? Does the algorithm work against those being evaluated? Can the algorithm scale (can it be deployed in large, technocratic structures)? If the answer to all three of these three questions is yes, then O'Neil argues that we should apply the WMD label.

A WMD performs an evaluation that cannot be seen by the people affected, is organized to benefit those who control the algorithm and/or the data, and can be applied across a large system or many systems that reach millions or billions of people. Welcome to the world of digital injustice.

Some industries, such as manufacturing, have found ways to build systems that account for failures in their technologies, and trigger warnings that corrections are needed. When electronic components like capacitors, electro-mechanical systems like airbags, or chemical systems like lithium batteries show evidence of failure, repair bulletins are triggered, recalls are made, or travel bans are put in place in order to keep us safe. Are similar mechanisms in place to guard against malignant algorithms that process data and shape human or automatic decision making? O'Neil's book invites us to ask: Is it time for us to deploy these safety mechanisms for the algorithms that affect our social lives?

While failed components are in a sense hidden until they set our phones or laptops on fire, once the problem is revealed the access to solutions or interventions is more direct. Robber baron algorithms may not be so easily ferreted out and avoided. Does a bank client have access to the guts of an application process? Certainly not. And when you are denied, how do you negotiate? On what grounds? Can you negotiate with a machine?

Welcome to the world of digital injustice

If a bank employee enters financial and personal information into a loan-application program at their desk and the client's loan application is turned down, are there any ways of evaluating how well the software evaluated that particular case? You could go to a different bank. Unless, of course, the problem is driven by a more basic algorithm such as your FICO-score calculation, which all banks will use. The loans are evaluated in aggregate, but the denial is experienced by the individual. Conversely, if the software approved a loan, how would we know in that case that the decision was a good one? Who evaluates the evaluating algorithms?

In the context of the ills that Cathy O'Neil has unveiled, it seems there are at least two very important hinge-points that require our attention: transparency and literacy.

Lifting the Hood

Transparency is the ability to see the rules that constitute the substance of an algorithm's decision-making process and the data that fuels those automated sorting and evaluation processes. This transparency is required for citizens to be literate about the ways their lives are disciplined by algorithms. In our era, data literacy—not only about data privacy but also about the ways in which data is generated, where it travels, and the algorithms it feeds—has become extremely important as a function of effective citizenship. We may not be able to stop exploitation when we see it, but we stand a better chance of doing so if it is visible to us. Take the case of the financial industry: You can see why certain forms of exploitation are kept out of view, hidden in complex fee agreements and penalty clauses. If the financial planner told us up front they what they were going to charge, many people would balk or walk.

Most of us are not the architects of mass digital destruction, but our habits and quiet compliance provide the ready conduits by which those weapons accumulate power.

George E.P. Box, one of the great statistical minds of the twentieth century, noted that "all models are wrong but some are useful." In order to understand the dynamics of algorithms, we need to think about and understand their usefulness. Box suggests that we evaluate our algorithms by asking three questions: Useful to whom? Useful for what? Evaluated how?

Over the years, Neil Postman, Jacques Ellul, Ivan Illich, and many others have reminded us that there are no neutral technologies. Change favours some and penalizes others, enriches along one line and impoverishes along others. Justice, love, and mercy are deeply woven into human civilization, sometimes taking the day, other times carrying on suppressed and subterranean. Rapid and socially pervasive machine-driven adoption of new technologies has, in some cases, outstripped our collective ability to bring justice, love, and mercy to bear on our inequalities. In many cases, we simply don't see it happening. The rapid adoption of social technologies provides one example, but as O'Neil outlines, unfair technologies are woven into education, the justice system, banking, and business. It would be worth exploring further how those who own these technological means and benefit from the data they extract are contributing to the gap between the haves and have-nots globally. This dynamic may be driving the acceleration of privilege for the powerful and the diminishing prospects for the weak. An algorithm doesn't work alone; it needs us as an accomplice. But when an algorithm finds the means, it dutifully pursues its logic without any ability to calculate the catastrophic effects on the powerless.

In an important 2011 article in Scientific American David Weinberger raised a challenging aspect of the data-processing deluge: What if an algorithm suggests a solution to a problem that is too complex for us to understand? Will we trust it? In the case of the American pilot featured in the movie Sully, the models said he could make it to a nearby airport. He didn't agree. His instinct, based on a career that included the safe delivery of more than a million passengers, took into consideration factors that the models did not. And he was right. What if a global policy supercomputer suggested the best way to take care of the tensions in the Middle East was to give every country nuclear capabilities? What if we couldn't explain why that made sense but that in feeding the supercomputer massive amounts of data, that was clearly the best solution?

The powerful have faithfully integrated into our technological systems our ages-old human propensity to exploit each other. We will not eliminate this. But we can push back on those inequalities, as we always have. Transparency grows with awareness. Literacy will also grow as we see, understand, and act to change the patterns of exploitation. This growth will require new tactics that are tuned to how the hidden, powerful, and pervasive systems work. It will require new legal and legislative powers that protect the freedom of citizens, not just those in power. Effective powers that rebalance inequalities can lead to greater trust and increase the beneficial potential of the data streams that grow daily. We will need to consider how the creativity of care could be fostered to outpace the creativity of domination.

Can we Turn Digital Weapons into Ploughshares and Pruning Hooks?

O'Neil at times sounds like Oppenheimer post–Manhattan Project: Having designed and built the Great Weapon, the nuclear scientists understood clearly that they could no longer function as value-free scientists who were just doing science without regard to how it would be used. Scientists have ethical responsibilities. Mathematicians, statisticians, data scientists, and quantitative analysts are responsible for thinking about how their powers may be deployed. Looked at in that way, Weapons of Math Destruction is the story of how a data scientist grew to understand the power of her craft not only in her hands but also in the hands of those employing her services.

Her awakening is also ours. We are all complicit to some degree in the unexamined adoption of so many socially invasive technologies. The cultural values that are necessary for WMDs to flourish are values we have largely adopted without question. We want information and entertainment, so we give up personal data for free. We want to feel connected and to belong, so we surrender our images, family exchanges, likes, and dislikes to massive data-collection machines that target our vulnerabilities and needs. Organizations, even small ones, accept process efficiency even when it erodes the culture of their work environment. Bankers, insurance agents, and business leaders forego the longer work entailed in actually knowing the people they serve and delegate relationships to automation. We like it too, because it is faster and cheaper. Why talk to a minimum-wage employee at McDonalds when you can order on a touch screen and eliminate the human? It's very efficient.

Most of us are not the architects of mass digital destruction, but our habits and quiet compliance provide the ready conduits by which those weapons accumulate power. There is the old vision of turning swords into ploughshares, of transforming destruction into generative work. We need a metaphor that captures how the tools of calculation, sorting, data processing, and digitally supported decision making can be developed toward socially generative ends.

O'Neil concludes with a review of the generative possibilities that are emerging, the ploughshares of the digital good. The long work of reforging digital weapons into tools stands before us to contemplate. If courage and intention meet, we may yet sow the seeds of generative common goods into the furrow of that digital soil.

Milton Friesen
Milton Friesen

Milton Friesen was Program Director, Social Cities for Cardus from 2008 to 2020.


Download and Share Articles From The Comment Reader

An introduction to Public Theology for the Common Good

Want more of the same fresh, thought-provoking content delivered right to your inbox once a week?