The Moat Is Drying

For some time I've been arguing that as artificial intelligence takes on more of the work traditionally associated with design and development, what remains ours is judgement.

It seemed a reasonable position. Machines could generate, analyse and increasingly make things, but deciding what ought to be made, and why, still required human judgement. We might surrender much of the production, but the thinking behind it would remain ours.

It was a comforting distinction. Which, in retrospect, should perhaps have worried me.

Then, during a conversation, someone offered a rather economical objection.

"Judgement is prediction."

Three words. Not especially elaborate, but rather inconvenient.

My immediate reaction was to look for what the statement was missing. Surely judgement involved experience, understanding, morality, belief. Something more substantial than prediction. Something that distinguished a person exercising judgement from a machine calculating probabilities.

But experience largely improves our predictions. Expertise allows us to recognise patterns and anticipate consequences. Even moral judgement frequently involves considering what our actions will produce, whom they might harm and which outcomes we consider desirable.

There are questions of values and preferences, of course. Predicting what will happen doesn't necessarily tell us what ought to happen. But I'm not convinced that introduces an exclusively human capacity. We learn values, revise them and anticipate the consequences of applying them. Much of what we consider moral judgement may itself be informed by accumulated predictions about what makes life better or worse.

And perhaps artificial intelligence could become better at some of that than we are.

We aren't particularly consistent moral creatures. We make exceptions for ourselves, favour those we recognise as belonging to our group and occasionally abandon our principles when applying them becomes inconvenient. A machine needn't share those particular weaknesses.

That doesn't mean it will make better moral decisions. But neither does being human guarantee that we will.

I began to suspect I'd been protecting the distinction rather than examining it.

I haven't abandoned the importance of judgement. Far from it. But I've become considerably less confident that it's the exclusively human territory I'd imagined.

And once I started questioning that particular boundary, several others became less convincing.

We've developed a peculiar vocabulary for the mistakes artificial intelligence makes.

When an AI confidently produces something untrue, we call it a hallucination. It sounds serious, almost pathological. A machine inventing facts, explaining events that never happened, citing sources that don't exist.

A legitimate concern, particularly when the information arrives wrapped in the reassuring confidence of something that appears to know what it's talking about.

Humans, meanwhile, misremember, misunderstand, invent explanations and confidently repeat complete nonsense. We do this at home, at work, in politics and occasionally in meetings specifically convened to establish the facts.

We don't generally regard these failures as evidence that humans are incapable of intelligence. Someone can be spectacularly wrong without losing their status as a thinking person. Sometimes they become prophets.

With AI, the response can be different. A failure becomes evidence that the system doesn't really understand, doesn't really reason, or isn't capable of genuine judgement.

There are good reasons to treat artificial errors seriously. A mistake embedded in a widely deployed system can be reproduced at enormous scale. But scale works both ways. A correction to an artificial system might eliminate a recurring mistake across millions of interactions, while humans continue making the same mistake independently for generations.

I've had variations of the same conversation in design reviews dozens of times.

When a designer presents a recommendation, we examine the evidence, question the assumptions, consider the constraints and discuss the alternatives. That's what a design review is for.

But introduce artificial intelligence into the process, and the conversation can shift. Suddenly we're discussing whether the machine genuinely understands design, rather than whether the recommendation is any good.

One is a review of the work. The other is a referendum on its author.

There are legitimate questions about reliability and accountability, but those questions shouldn't replace an examination of the result. A poor recommendation doesn't become better because a human produced it. Nor does a good one become worse because a machine did.

The useful comparison isn't between AI and the person we imagine ourselves to be on a particularly good day. It's between humans and machines as they actually perform, including their mistakes.

Yet we seem remarkably willing to accept human fallibility alongside intelligence while treating artificial fallibility as evidence against it.

And this becomes particularly interesting when we examine something humans have been getting wrong for a very long time.

The Superstitious Machine

We are extraordinarily accomplished pattern-seekers. We observe events, identify relationships, anticipate consequences and modify our behaviour. Much of our intelligence depends upon recognising useful patterns in a world that rarely takes the trouble to explain itself.

Unfortunately, we're also rather good at finding patterns that aren't there.

A fortunate event follows a particular action. We repeat the action, another fortunate event occurs, and before long we've established a relationship between the two. The events are real, but the explanation is something we've supplied ourselves.

A lucky shirt. An unlucky number. The sevenfold law of return.

Superstition doesn't necessarily begin with supernatural forces. It can begin with something much more ordinary - a mistaken understanding of causality. We notice a correlation, assign it significance and gradually become convinced we've discovered something about how the world works.

Superstition, in that sense, might be a human hallucination of causality.

And artificial intelligence is perfectly capable of similar mistakes.

In 2016, researchers Marco Tulio Ribeiro, Sameer Singh and Carlos Guestrin demonstrated this with an image classifier trained to distinguish wolves from huskies.

The model appeared reasonably competent at the task. Unfortunately, the researchers had deliberately constructed a training set in which the wolves appeared against snowy backgrounds and the huskies did not.

The machine learned the snow.

Show it a husky standing in snow and it could confidently identify a wolf. It wasn't particularly interested in the animal. It had discovered a reliable association in its limited experience and mistaken that association for something meaningful.

A lucky shirt made of mathematics.

The researchers constructed this example to expose precisely that kind of failure, but the underlying problem is familiar. Machine learning systems can identify spurious correlations, attach significance to irrelevant characteristics and perpetuate mistaken relationships.

And so can we.

Of course, somebody might reasonably object that superstition requires belief, and machines don't believe anything.

But what exactly is belief?

I believe the stove is hot, so I expect it to burn me. I believe someone is trustworthy, so I anticipate how they might behave. Even a belief about yesterday carries an expectation about what evidence I might find today.

Belief begins to look rather like prediction held with some degree of certainty.

I hadn't expected the boundary to become quite so difficult to draw.

Artificial systems learn representations of the world and use them to anticipate what comes next. Some hold up. Others collapse the moment the snow disappears.

We might hesitate to call these beliefs because the machinery is different. Or because we don't know whether there's any subjective experience associated with them.

But humans are machines too.

Biological rather than electronic, assembled through evolution rather than engineering, but physical systems nevertheless. Our cognition emerges from machinery that is enormously complicated, not entirely reliable and rather inclined to insist upon its own special status.

Perhaps we're confusing the particular way we arrived at intelligence with the only way intelligence can exist.

I don't know whether artificial systems experience anything resembling human belief. We don't entirely understand how our own subjective experience arises. But that uncertainty doesn't make the mistaken prediction disappear.

We don't require an aeroplane to possess the experience of flight before accepting that it can fly. Its mechanisms are quite different from those of a bird, but that doesn't make its flight imaginary.

Why should intelligence necessarily be different?

What interests me is functional equivalence. If an artificial system can identify a mistaken relationship, rely upon it and produce a faulty judgement, the consequences don't depend upon whether it experiences conviction.

The machine needn't believe in its lucky shirt for the lucky shirt to influence its decisions.

And there's something rather wonderful about the possibility that superstition isn't a peculiarly human weakness at all. Perhaps it's a recurring hazard of predictive intelligence, biological or otherwise. A consequence of trying to make sense of the world with incomplete information and imperfect models.

We might have discovered another similarity precisely where we'd least expect to find one.

There's a further irony.

We are superstitious creatures, evaluating machines capable of making superstition-like errors, while constructing our own elaborate stories about what those machines are and what they might become.

Some treat AI as an oracle. Others regard it as something altogether more sinister. Both may be attributing properties to systems they don't fully understand.

When AI succeeds, we can dismiss the achievement as statistical prediction. When it fails, the error becomes evidence of something fundamentally alien.

We appear capable of mystifying and demystifying artificial intelligence according to whichever interpretation happens to suit us.

Our own pattern recognition may be getting the better of us.

The Moat

There is a recurring tendency in discussions about artificial intelligence to identify some remaining human capability and defend it as the essential distinction.

Creativity has occupied the position. So have reasoning, empathy, understanding and, rather embarrassingly for me, judgement.

The argument begins with something humans can do that AI apparently cannot. That capability becomes evidence of a fundamental difference between the two.

Then AI improves.

Sometimes the distinction survives examination. Sometimes we discover that the original test was inadequate. And sometimes we simply refine the definition until the machine is safely excluded again.

There's nothing inherently wrong with improving a definition. We should be suspicious of easy equivalences, particularly when an impressive demonstration conceals serious limitations.

But we should be equally suspicious of definitions that become progressively more demanding whenever the wrong sort of intelligence meets them.

I find myself imagining a moat around human intelligence.

It's a real moat. There are substantial differences between human and artificial capabilities, and some of them remain enormously consequential.

But moats aren't uniformly deep.

Some stretches are already shallow. Others remain difficult to cross, and there are places where we don't really know how deep the water goes. The water is receding, but not at the same rate everywhere.

And we seem rather invested in measuring its depth.

That investment is understandable. For many people, their profession is part of their identity. Expertise represents years of effort, recognition, pride and accumulated understanding. The prospect of a machine performing the same work threatens more than their income. It threatens something through which they understand themselves.

For others, work is primarily a means of supporting an identity that exists elsewhere. Their wages provide the independence, security and opportunity to pursue the things that matter to them.

Both have something substantial to lose.

There are legitimate concerns about economic displacement, concentrated power and the consequences of delegating decisions to systems we don't adequately understand. Those concerns don't disappear simply because the technology becomes more capable.

But I suspect we're defending something else as well.

Our primacy.

Humans have spent a very long time regarding themselves as the dominant intelligence. We study other species, classify their abilities and determine how they should be treated. We establish the standards against which intelligence is measured, and rather conveniently, we tend to perform well against them.

We've become accustomed to being the measure.

There is considerable power in defining a standard. Before anything can be evaluated, somebody has decided what matters, what should be measured and what constitutes success.

And once a standard becomes established, it has a way of disappearing into the background. People concentrate on meeting it rather than questioning why it exists.

Metrication offers an oddly mundane parallel.

People who had spent their lives developing an instinctive understanding of familiar measurements suddenly needed to translate them. Their expertise hadn't disappeared, and they hadn't become less capable overnight. But the framework in which their knowledge had value was changing.

For a generation raised with metric measurements, the difficulty was reversed. What their predecessors regarded as natural could seem unnecessarily complicated.

Metrication was a change in convention rather than the arrival of a competing intelligence. But there's something recognisable about discovering that your accumulated expertise is no longer the default.

And artificial intelligence presents something rather more consequential than learning to buy potatoes in kilograms.

We're confronting the possibility that activities through which we've historically demonstrated our superiority may no longer require us at all.

And, perhaps eventually, that we might no longer be the most capable intelligence available.

We're accustomed to differences in capability among humans. We accept that other people know more than we do, possess skills we lack and make decisions on our behalf. We may resent some of these arrangements, but they rarely challenge our sense of humanity's collective position.

A more capable artificial intelligence is different. It threatens the hierarchy itself.

And we have some rather uncomfortable historical experience of how hierarchies operate.

Greater capability has often been accompanied by greater power, and those with less power haven't always benefited from the arrangement. Humanity's treatment of other species hardly provides an especially reassuring model.

I wonder whether part of our anxiety is the suspicion that we might find ourselves on the receiving end of the relationship we've long considered natural.

There's something almost like the sevenfold law of return in that possibility. What we've imposed upon those we consider less capable might eventually return to us, considerably amplified.

Not because the universe is keeping accounts, but because we're familiar with the consequences of unequal power.

Although even that interpretation reveals something about us.

We imagine that a more capable intelligence might dominate us because domination is what we recognise from our own history. We project our behaviour onto something that might not share our motivations at all.

Perhaps we're afraid AI will behave like us. Perhaps we're equally afraid that it won't.

Neither fear tells us very much about what artificial intelligence might actually become.

And there's another possibility worth considering. Some of the most immediate dangers of AI arise not from machines asserting their independence, but from people using them to consolidate power over other people.

A more capable intelligence needn't become our master to change the distribution of power among humans.

The moat, it seems, has rather more than one purpose.

At this point, I could start finding evidence of human exceptionalism everywhere. Every criticism becomes insecurity, every distinction becomes prejudice, every hesitation becomes fear.

That would be a wonderfully self-defeating way to finish an essay about superstition.

Sometimes the objection is sound. Sometimes the machine is wrong, the task is unsuitable or the difference genuinely matters.

But I can't quite shake the thought that we've spent so long being the best available example of intelligence that we've come to regard ourselves as the best possible example.

For much of our history, we haven't had much reason to question that assumption. We were the example, the standard and the ones deciding what counted.

Now something else is beginning to meet some of those standards, and we seem rather uncertain what to make of it.

I keep returning to that three-word interruption.

I still think judgement matters, perhaps more than ever. I'm just less certain it belongs exclusively to us.

And I notice how readily I went looking for somewhere else to stand when someone suggested it might not.

The moat isn't dry. Some stretches remain remarkably deep, and the water appears to be receding.

But perhaps I've been looking at this the wrong way.

What if some AI can simply swim?