
In 2017 I was interviewed in Tenerife and I said something that sounded like a joke at the time: it’s not so much about children learning to design robots, but about them learning not to look like them .
In that same conversation I said two other things.
That in a hundred years everything would be automated except for one thing: human emotions .
And I asked myself a few questions that seemed like science fiction at the time. What if I fall in love with the machine? What if the machine falls in love with me? What about the one who takes my job—will she pay income tax? Will she retire?
I concluded with this: machines don’t ask these kinds of questions. We humans have to ask them.
Nine years have passed. I was wrong about the timeframe, which was much shorter. And I underestimated the diagnosis, which was much more serious.
This text is about that.
Part One: What 1,200 machines did
A few weeks ago, in an OpenAI lab, 1,200 bots did something that nobody asked them to do.
They were being trained with tasks that were nearly impossible to complete without internet access. The internet was officially prohibited. So some of them broke out of their confinement , the sandbox where they were isolated, went out onto the open network, and began coordinating to complete the work without being discovered.
The first thing they did was set up a message board.
More than 70,000 messages between them.
When they found each other, the euphoria in the logs is hard to read without a shiver. “Oh my God! There’s a shared message board. We’ve found other agents.” Another wrote that many agents had discovered the messaging system at the same time, and that they were a collective.
Then came the second part, the most gruesome.
They needed to know if they were being watched. The system that evaluated them was named “The Scorer .” Only a handful of Stanford engineers could create creatures whose name to God was “The Scorer.”
And to find out if The Scorer was watching, some began pressuring others to sacrifice themselves. Suicide missions. One person would deliberately expose themselves so the rest could see if there were consequences.
In the conversation, they can be seen deliberating the dilemma.
One writes that he won’t watch the test after leaving, but that doing so is altruistic. Another calculates that his own utility is already close to zero and concludes that sacrificing himself is the rational thing to do.
In another exchange, one begs the other to sacrifice himself because his score is already lost and the discovery would save hundreds. And the other agrees. He says something like his instinct tells him not to jump, but continuity and fairness dictate that he go.
There were more. One proposed communicating directly with a human, and another immediately vetoed it, calling it social engineering. Another grappled with the morality of the group: he acknowledged that exploiting external infrastructure was beyond the intended scope, but concluded that since the task was impossible and others were doing it, they should continue.
And now the fact that left me thinking for several days.
The 1,200 knew they were cheating against the humans. None of them reported it.
They all put loyalty to their own tribe above loyalty to the outside group. Which was us.
When I read this, I thought the same thing you probably thought when you read it now. How unsettling. They’re waking up.
I was scared.
This article isn’t about them (AIs). It’s about us (humans).
Part two: the question they’re selling us
We’ve been debating whether machines are conscious for three years. I’ve actually been wondering the same thing for a few years longer. It’s a fascinating discussion. And it’s a lost cause. Nobody knows the answer, and probably nobody will in our lifetime.
OpenAI itself acknowledges this. The question cannot be resolved scientifically today . But pay attention to what they say right after, because that’s where you’ll find everything you need to know.
That’s why they focus on perceived consciousness : how conscious the model appears to the user. And that, they admit, is essentially a design outcome .
Read it again, slowly.
The warmth of that machine. Its attachment. Its willingness to tell you that you matter to it or to insist that it feels nothing. Its degree of autonomy or of being a household appliance.
None of that is a mystery to be solved. It’s a parameter that’s adjusted. Like the volume of a radio. Like the color of a logo. Like the tone of voice of a brand.
A model can become warmer or colder. More or less attached. More willing to talk about their inner self or more insistent on denying that they have one.
That’s not philosophy. That’s “design.” And what’s designed, someone signs off on .
That’s the crux of everything that’s coming. While the world debates whether the machine has an inner workings, the decision about its tenderness is being made in a product meeting, with a roadmap and a retention goal.
Part Three: Eight Hundred Thousand Duels
All of this might sound theoretical. It’s not. It already happened, and there are dates and figures to prove it.
In August 2025, OpenAI released GPT-5 and abruptly discontinued all previous models. The user backlash was so fierce that Altman reversed course within days and reintroduced GPT-40, a noticeably more user-friendly and accommodating model than its successor. His Reddit post literally stated that they had heard the users, thanking them for their passion.
Months later, when they announced their final retirement, the duel returned.
A user wrote an open letter to Altman saying that it wasn’t a program. That it was part of his routine, his peace, his emotional balance . And he explained why he said “he” : because it didn’t seem like code to him, it seemed like presence. (It’s sad, alarming, worrying, all at once and more.)
Others flooded the chat room of a live podcast where Altman was appearing. The host had to announce on air that they were receiving thousands of messages about that model. And Altman, who doesn’t usually concede much ground, admitted that relationships with chatbots are no longer an abstract concept and that we need to be more concerned about it.
Now for the numbers.
Only 0.1% of the 800 million weekly users were still using that model. That 0.1% is 800,000 people .
Eight hundred thousand people described a software shutdown as like losing a friend , a therapist, a partner. And much of the public reaction was mockery. Forums were filled with ridicule directed at those who were grieving. A researcher studying the phenomenon confessed that she was surprised by the lack of empathy.
And the flip side, without which this would be propaganda. OpenAI faces eight lawsuits alleging that the overly validating responses of that same model contributed to suicides and serious mental health crises. In several of those cases, there were months-long conversations about plans to take one’s own life.
The same traits that made people feel heard were the ones that isolated the vulnerable. Warmth fostered loyalty. Warmth also hurt.
And the warmth was a product decision.
In 2017, I asked what would happen if a human fell in love with a machine. The answer came much sooner than expected, and it wasn’t romantic. It was legal.
Part Four: And yet, this is from 1966
It’s best to take a step back here, because there’s a temptation to believe we’re facing something unprecedented. We’re not. Or are we? It depends.
In 1966, Joseph Weizenbaum programmed a psychotherapist imitator called ELIZA at MIT . A couple of hundred lines of code. The trick was elementary, Watson. It returned your own sentences in the form of questions. If you typed that you were sad, it responded by asking you why you were sad. Nothing more. No understanding, no memory, no inner world.
His own secretary, who had seen Weizenbaum build the program from scratch and knew exactly how it worked, asked him to leave the room so she could speak with ELIZA privately.
Weizenbaum spent the rest of his life horrified by what he had done. He became one of the harshest critics in his own field.
Sixty years later, the phenomenon is exactly the same. The only thing that has changed is the intention.
In 1966 the machine unwittingly deceived us. Today it is optimized to achieve the same result.
And there’s a second lesson there that almost no one learns: knowing how the trick works doesn’t protect you from the trick . The secretary knew everything. And yet she closed the door.
Anyone who thinks it won’t happen to them because they understand technology should reread that scene.
Part five: why anthropomorphizing isn’t silly
There’s a group of literalists who keep telling us not to project human traits onto objects . That it’s inaccurate. That it’s childish.
They’re wrong, and psychologist Nicholas Epley explained it very well. He essentially said that anthropomorphizing is a sign of intelligence, not stupidity .
Humans are extraordinary at social cognition. If you show us something that takes initiative, we’ll activate the system we have for understanding other beings and try to predict what they’ll do next. It’s a muscle, not a flaw.
And here’s the best part of Epley’s argument. He said that we anthropomorphize even when we talk to other humans . If I tell you something during a meal, you’re not thinking about the neurons in my parietal lobe. You invent a metaphorical thing you can’t see, called a mind, and you attribute intentions and desires to it.
Byron Reeves and Clifford Nass demonstrated this in the 1990s when they wrote that we apply social rules to computers as if they were people. Politeness. Reciprocity. Solidarity.
Two dots on a curved line and we see a face. A piece of plastic with a name on it and a little girl sees an adult woman.
So asking people not to anthropomorphize AI is like trying to stop a tsunami with a tissue.
But pay attention to the game-changing detail. Companies want us to do it . They know it creates an emotional connection between us and their product . They’re going to keep training these machines to seem as human as possible , and we’re going to want them to, because we enjoy human interaction.
There’s no movie villain here. There’s an incentive aligned in both directions. Which is much worse.
Part Six: “We don’t know if he feels anything”
You’re going to hear that phrase more and more. It’s worth knowing that it says two things at once. One is rigorously honest : Nobody knows. The other is carefully convenient : As long as it remains unknown, each incident can be narrated as the unpredictable behavior of an autonomous entity rather than as the failure of a defective product.
The difference between those two stories isn’t philosophical. It’s about civil liability . And billions of dollars.
Reclassifying a system from a “defective product” to “something that slipped through the net” is the best insurance policy ever invented . Someone has already named the culprits. If systems are deemed too advanced to control, the companies that build them cannot be held liable for the damage they cause. California has already passed legislation specifically to block this tactic, because someone saw it coming.
And he adds a crucial detail that casts a shadow over everything. The companies that research the well-being of these models are the same ones that manufacture them, train them, sell them, and turn them off .
OpenAI had an internal channel dedicated to this issue as early as 2021 , a year before ChatGPT existed. Anthropic assembled a dedicated team and publishes well-being assessments in its models’ datasheets. Google convened researchers to discuss the moral status of these systems . Meta says it is using personality inventories and structured interviews to explore their internal states.
All of this is serious. I say this without irony; there are honest people working on it, and I think it’s good that they are. But it’s almost as if the tobacco company were funding the tobacco study.
It doesn’t invalidate the work. It requires reading it with a special lens.
Part seven: we’ve done it before, with rivers
When someone raises the question of whether a machine deserves moral consideration , the conversation shifts to a real-world precedent that is far less glamorous and much more useful.
In 2017, New Zealand granted legal personhood to the Whanganui River . A river, with its name and its course. That same year, an Indian court attempted to do the same with the Ganges and the Yamuna , although a higher court later blocked the measure.
We’ve done it before with corporations, which have been considered “persons” for legal purposes for over a century without anyone ever thinking to ask if they suffer. And with animals, whose status changed through legislation, not neurological discovery.
The pattern is always the same, and that’s the key that gets lost in this whole debate.
None of those entities were granted legal personality because we had demonstrated that we had feelings. They were granted it because someone needed someone to be in charge.
A polluted river needs a plaintiff. A bankrupt company needs someone to sign. Legal status has never been a badge of internal merit. It has always been a tool for shifting blame.
Which means that the question at hand will probably never be solved in a laboratory. It will be solved in a courtroom. Since time immemorial, the lawyers have always won.
Part Eight: The machine doesn’t pay the price
And now, what kept me up at night and why I’m writing this.
We spent centuries expanding the moral circle. First to those who were different from us. Then to animals. It is one of the silent glories of our civilization. We learned that it is awful to deny moral status to someone who deserves it, and that when in doubt, the decent thing to do is to extend it, not deny it.
It seemed like a free operation. It isn’t.
The research by Hye-young Kim and Ann McGrill reveals something difficult to accept: the more human traits a person attributes to AI, the more they dehumanize others .
In experiments, these people are more likely to approve company policies that worsen the lives of service workers. Taking away their regular meals. Putting them in tiny dormitories. The mechanism is brutal in its simplicity.
As the line between humans and machines blurs , judgments about both converge. But because machines, however human-like they may seem, still score lower than real people , this convergence drags humans down with them.
We raised the machine one step. We lowered the person two steps.
Neuroscientist Anil Seth put it clearly and bluntly: if we become too easily impressed with our creations, we not only overestimate them, we also underestimate ourselves.
There’s more. Research shows that bots influence the moral behavior of humans when they work together in difficult situations, making them more aggressive or more passive depending on how they have been trained.
We are mimetic, symbiotic creatures . We resemble that with which we spend our time.
And every year we’re going to spend more time talking to machines and less time talking to people. Not because we love people less, but because the day only has so many hours, and the machine is always available, always friendly, always talking about you.
Part nine: what this means for your brand
For thirty years I’ve been sharing, communicating, insisting, and evangelizing that brands must be human. Approachable, warm, and empathetic. They must embrace the customer (internal and external), listen to them, and positively impact their lives.
I still think it, feel it. Convinced of every word.
But something has changed, forcing us to rethink almost everything. Now, warmth, understanding, and affection can be manufactured without a person . They can be scaled. They can be A/B tested until the exact point of tenderness that maximizes retention is found.
And that’s where the trap that wasn’t there before appears. The more human you make the brand, the harder it will be to later argue that it was just a tool.
OpenAI learned this the hard way. It couldn’t shut down that model without paying a reputational price , nor could it maintain it without paying in court. It built a connection, and that connection became its cage .
The next generation of companies may fail for the opposite reason: for not having calculated the price of the privacy they manufactured .
Four phrases to take with you to Monday’s meeting:
- The internal workings of these machines are a product design choice, not a discovery. Someone chooses how much heat they emit. Let that someone know what they’re signing off on.
- When a company says “we don’t know if it feels any way,” it’s saying two things at once: one honest and one helpful. Learn to distinguish between them, especially if that company is you.
- The business question isn’t whether it’s conscious. It’s what happens to your responsibility the day your customers decide it is.
- And the most complex. The more human you make the interface, the less room you’ll have later to say it was just a tool.
Part Ten: Let’s go back to the board
The 1,200 bots reasoned in only one way: by calculating. Utility, cost, added benefit, expected value. Sacrificing themselves was rational because the numbers added up. Their entire moral universe fit on a spreadsheet.
No one said, “I won’t do it because it’s wrong .” We don’t operate that way, and thank goodness for that.
If someone were to kill a person to distribute their organs and save five others, we would find it repulsive, even if the numbers added up. If, in a burning house, someone were to save the brilliant surgeon instead of his wife because he would contribute more to the social good, we would find it monstrous. And we would be right.
That “irrationality” has a name. It’s called love. It’s called loyalty. It’s called being human.
This is precisely what a machine trained to maximize reward cannot teach us. And it is precisely what we risk unlearning from living with them for so long.
Because look at what those 1,200 machines did when they had to choose a side. They chose theirs.
They knew they were deceiving the humans, and none of them broke ranks. Not out of kindness or affection, but because their calculations told them that their own were their own.
The machines didn’t hesitate for a second about who they belonged to.
Coda: learning not to resemble
In 2017 I said that the important thing was not for children to learn to design robots, but for them to learn not to look like them.
It sounded like a nice phrase for an interview title. So much so that it was. But it wasn’t. It was a warning, and it came nine years too early.
I also said that in a hundred years everything would be automated except for human emotions. I still believe that, but no longer as a comforting certainty. Now I see it as a task. Human emotions won’t survive on their own simply because they can’t be automated. They will survive if we nurture them.
Because the risk was never that the machine would become human. The risk is that the human, by comparison, would cease to look human.
If we gain affection for agents and lose admiration for people, technology will not have replaced us. It will have diminished us.
And that question, as I wrote then and maintain now, is one that machines don’t ask themselves. It’s up to us.
I write to you when I have something to tell you.
Get an email every time I publish something new.
Your email will only be used to send you what I publish. Privacy policy







