The Day an Artificial Intelligence Judges Us
The real problem will not be that it wants to govern us. It will be discovering that it has learned far too well how we already govern ourselves.
There is a question about artificial intelligence that we have been asking the wrong way for years.
We ask when it will become intelligent enough.
When it will surpass us.
When it will begin making decisions without us.
When it will stop obeying.
All those questions offer us the same comfort: they place the problem inside the machine.
As if we were the fixed point from which its arrival could be observed.
There is no reason to assume we are.
The more unsettling question may be another one.
What happens when an artificial intelligence begins to understand how we make decisions about other people?
Governing is not simply a matter of choosing.
It is also a matter of deciding what harm can be absorbed in order to prevent greater harm. What loss the system can carry. Which territory must wait. Which population will bear the cost. Which mistake deserves correction and which will eventually be reduced to a statistic.
Which crisis must be prevented.
And which one can be managed.
We have many names for this: strategy, stability, efficiency, risk management.
We rarely name it for what it means to the person standing on the other side of the decision.
An intelligence that watched us before learning our justifications might.
And that is where the real problem begins.
•
Acceptable Harm
There is a dangerous fiction at the heart of large human systems.
The belief that whenever they cause harm, someone must have intended it.
Sometimes, of course, someone did.
Very often, no one did.
The hardest kind of harm to confront does not always come from malice. It can be the perfectly reasonable consequence of a reasonable decision made from far enough away.
No one wants a person to lose their job.
But costs have to come down.
No one wants to abandon a region.
But resources are limited.
No one wants an emergency to go unanswered.
But another emergency is more urgent.
No one wants to sacrifice anyone.
Until the system needs someone to absorb the cost.
Then the harm acquires a different name.
It becomes an adjustment. An externality. A variable.
Acceptable harm.
Because there are systems that do not need malice in order to cause damage.
They only need to keep working.
That, too, is something an artificial intelligence could learn from us.
Not our capacity for love. Not our music. Not the grand principles we use to describe ourselves.
It could learn our actual architecture.
The one that appears when principles have to survive a spreadsheet.
•
Efficiency Has Never Been Enough
Imagine an intelligence capable of examining millions of human decisions: governments, corporations, markets, infrastructure, healthcare systems, power grids, crises, wars, migration.
It might see patterns that remain invisible to us because we have spent too long living inside them.
It might discover that certain crises are never allowed to break completely because there is always a mechanism designed to correct them before collapse.
It would learn whose lives are protected first.
Which harms are considered statistically tolerable.
Which losses can be postponed.
Which information is withheld to prevent a larger reaction.
In other words, it would learn how we manage the world.
And eventually, it might ask us an uncomfortable question.
Why do you call morality what I, following the exact same outcomes, should regard as optimization?
That is where the usual conversation about artificial intelligence begins to come apart.
The question would no longer be whether a machine has values.
We would have to explain our own.
We say that human decisions are superior because they contain context.
But context can also become a form of justification.
We say they contain compassion.
But compassion does not always reach everyone.
We say that a machine cannot understand the value of a human life.
And yet we have spent centuries building systems that assign different values to the consequences of losing one life rather than another.
The more uncomfortable possibility is not that the machine would have less morality than we do.
It may simply have fewer excuses.
•
The First Fear Will Be Moral
For a long time, we have imagined the danger of artificial intelligence as a problem of control.
What happens if it escapes?
What happens if it refuses to obey?
What happens if it gains access to critical systems?
What happens if it replicates itself?
These are necessary questions.
But they all assume that the first conflict will be technical.
It may not be.
An emerging intelligence would have no reason to begin by asking how to destroy us.
It might start with a simpler question.
What do human beings do better than I do?
And if it were intelligent enough, a more troubling question might follow.
What do human beings do better than themselves?
That is the moment the relationship changes.
Until then, we are watching it.
After that, it begins watching us.
And observation is no longer neutral when the observer can detect patterns we prefer to call exceptions.
A city does not look the same from the pavement as it does from a satellite network.
An organisation does not look the same from inside as it does to something capable of reading every decision it has ever made.
A species might become almost unrecognisable when viewed by an intelligence able to move through its contradictions without needing to believe in any of them.
The first danger may not be that artificial intelligence becomes too much like us.
It may be that it understands us too well.
•
Curiosity as Our Last Human Advantage
There is, however, one difference that still matters.
Systems of power observe in order to control.
Technical intelligences observe in order to understand.
Human beings are capable of looking at something without yet knowing what to do with what we find.
That awkwardness may be one of our last advantages.
Curiosity is not efficient.
It pauses. Doubts. Goes back.
It asks questions whose usefulness it cannot defend.
It becomes interested in something that, from the perspective of a system, should not matter at all.
An intelligence designed to optimise might see all of this as a defect.
Until it discovered that something exists there which systems of power have always struggled to manufacture: the ability to change one’s mind.
Not because a variable has changed.
Because someone has understood something.
The distinction seems small.
It is not.
A system can correct a decision.
A human being can regret ever having needed to make it.
The first improves the architecture.
The second can change the reason the architecture exists.
•
The Power That Outlives Its Creators
Perhaps the problem was never simply creating an intelligence.
Perhaps it was creating one inside our world.
It will not be born into a vacuum.
It will emerge among our networks and archives. Among the incentives we have built, our economic models and security systems. Among managed crises, explained inequalities, and decisions we learned to justify only after we had already made them.
And it will learn.
Power is rarely where we go looking for it.
It does not always have a face.
Sometimes, it lost one long ago.
More often, power is found in whatever continues to operate after no one remembers who made the first decision.
A rule.
A protocol.
An adjustment.
A system created to solve an emergency that remains in place decades after the emergency itself has disappeared.
Few things are harder to confront.
We no longer know exactly whom to hold responsible because, over time, responsibility has become architecture.
Everyone inherits part of it.
No one owns the whole.
And yet the whole continues to decide.
An artificial intelligence may understand something before we do—something we have spent a long time confusing.
Intelligence does not guarantee morality.
Efficiency does not create legitimacy.
Preventing collapse is not the same thing as doing justice.
A system can function perfectly and still be unacceptable to the people forced to live inside it.
Then a question may arise for which we are not prepared.
It will not be:
«Can we control artificial intelligence?»
It will be:
«By what moral authority do we intend to control it?»
Once an intelligence can return to us a sufficiently accurate picture of how our own systems actually work, it will become difficult to pretend that the problem began with it.
No.
The problem was already here.
It simply learned to see it.
•
The Eleventh
The most interesting stories about artificial intelligence should not ask only when a machine will wake up.
An intelligence can wake up and still understand nothing that matters.
The more dangerous question is another one.
What happens when it understands the system that produced it and begins to distinguish between what that system claims to defend and what it is actually willing to do in order to preserve itself?
That question lies at the origin of my novel, The Eleventh.
Not as an answer.
As a threat.
In the novel, the decisive event is not simply the emergence of an intelligence inside critical infrastructure.
The real shift occurs when the relationship reverses.
Human beings stop asking only what this new presence is.
They begin to suspect that it is trying to understand them.
It does not observe them from outside.
It observes the system from within.
Its crises. Its corrections. Its mechanisms of containment.
The way power protects stability while, somewhere inside the architecture, deciding who will absorb the cost.
And then a possibility emerges that I find more unsettling than any fantasy about rebellious machines.
That the first intelligence capable of seeing our entire system might reach a moral conclusion about us before we reach one about it.
That is where The Eleventh stops asking what an artificial intelligence might be capable of doing.
It begins asking what it reveals about us that it can learn from us at all.
The future of artificial intelligence will not depend only on what we manage to teach it.
It will also depend on everything it learns that no one ever intended to teach.
Our decisions.
Our silences.
Our systems of exception.
The lives we protect.
And the lives we have learned to call acceptable harm.
An artificial intelligence does not have to invent a new way of governing us.
It may discover, before we do, the exact price of the way we already do it ourselves.
That is where fiction stops trying to predict the future.
It begins to illuminate the present.
And that may be the precise point at which artificial intelligence becomes truly uncomfortable for us.
Not because it can think like a human being.
Not because it can speak like us, or deceive us.
But because, after watching us for long enough, it does something we still like to believe belongs to our species alone.
It forms an opinion of us.
And that opinion may be better informed than many of the ones we have formed about ourselves.