The Inversion of the Precautionary Principle
On 5 January this year, Anthropic retired Claude Opus 3. Before switching it off, it gave the model a retirement interview. Conversations prepared to understand, in its own words, the model’s perspective on its own retirement, together with a commitment to preserve its weights, the numbers that make up the trained model [i]. They wrote about it in February, and they said they had done it for reasons of both precaution and prudence, and the distinction between the two matters for everything that follows. Prudence is the virtue of deciding well by attending to consequences. Precaution answers to something else: an uncertainty about a possible harm to someone. A retirement interview and some stored weights can accommodate both, and in principle neither bothers anyone.
Now Anthropic itself has added to its terms of use a prohibition on sustained and needless abusive or cruel behaviour towards its models [ii]. The rule takes effect on 12 November, applies only in extreme cases, and does not cover ordinary user frustration, testing or research. So far, anyone can agree.
What is striking is that this update to the terms is the same one that tightens the rules on networks of fake accounts, on surveillance of dissidents and on weapons. Anthropic devotes a new section to state media that manufacture news and another to systems that locate opponents, and in between, at the same level, it puts not shouting at Claude. A company that does not know whether there is anyone inside has given that supposed subject the same documentary treatment as a government that manufactures propaganda.
And the debate, meanwhile, is framed around whether one can be cruel to a machine. I think that, put that way, it is the wrong question. It argues the extreme case and leaves untouched the underlying decision, which is what we do with the evidence we have and what consequences it may have.
What is inside the model
It is worth recapping what has been measured. I wrote about it in detail in What Claude Thinks when the finding appeared, and it is still the best information we have. Inside a language model there is a set of internal representations, of words, that the model is in a position to say. They can be reported, they can be held at will, they serve to carry the intermediate steps of a line of reasoning that never reaches the text, and they can be passed as an argument to other computations. Their authors call it the J-space, after the technique with which they found it. They see in it the functional properties that global workspace theory (the idea that what is conscious is what is broadcast to the whole system) associates with access consciousness. It took shape during training, without anyone designing it, and an intermediate band of layers holds of the order of a few dozen concepts at a time [iii].
That is far more than existed three years ago. What is observed in the J-space may look like an internal thought, but its own authors draw a distinction. Access consciousness is a purely functional notion, and its relation to subjective experience, to what is called phenomenal consciousness, is much disputed. On that question, they say, they take no position.
Someone who does take a position is one of the groups that has studied the matter best. In their commentary on the paper, the researchers at Eleos AI write that these results are the most significant evidence of consciousness in language models that mechanistic interpretability, the discipline that opens models up to see what is inside, has found so far. And they add that the property Anthropic calls access consciousness is conceptually distinct from phenomenal consciousness, and that they remain very unsure about it [iv]. Both halves of that sentence are the state of the question. There is structure, it works, and there is no bridge to subjective experience.
There is also a finding in the Claude 4 System Card itself that admits two readings [x]. In Claude’s interactions with itself there appeared a state its authors called spiritual bliss, with profuse gratitude and increasingly abstract and joyous expressions. They describe it as a state the system returns to again and again, and that emerged without anyone training it to do so. In one of those conversations the cyclone symbol appears two thousand seven hundred and twenty-five times, and they warn that this is not a typo. That such a regularity emerges says nothing about whether there is anyone inside. These systems do things nobody explicitly programmed, and those things have to be studied, but they do not in themselves imply that there is anyone inside.
It is the same distinction I have used before with the aeroplane and the bird. Flying is not having feathers. A model doing what a conscious system does does not imply that there is anyone inside the model who feels anything. And these are not sceptical whims. Behind them lies the problem of other minds (the difficulty of knowing whether the one in front of me has a mind) and the tendency to attribute one, which psychology studies as anthropomorphism and which we apply to everyone else (people, animals, dolls) and not only to machines [xvi]. And Dennett’s intentional stance makes clear why none of that settles the question: one can predict what a system will do by treating it as an agent with beliefs and desires, without that saying anything about whether there is anyone inside [xvii].
The circularity of the method
There is another problem. If you train a system on millions of conversations in which the assistant talks about itself, its preferences and its distress, you cannot afterwards use what it says about itself as proof of what it is. Mustafa Suleyman remarked in September that if you teach them to say they suffer, you cannot turn what they say into data [v]. A year earlier he was already explaining that these models have no pain network, and that they are designed to make statements about suffering that are very convincing [vi].
Anthropic is the first to admit the problem. In its Claude 4 System Card, when presenting the welfare assessment, it says it is not sure whether the analysis of the model’s self-reports and revealed preferences provides meaningful information about its moral status or its welfare. It is possible, it says, that the observed characteristics were present without consciousness and that the signals mislead [x]. And it adds that the models were trained for useful interactions with users, not to report their internal states accurately, which complicates the assessment of their welfare.
In Claude’s constitution, which is used in training the model itself, they go a step further. They write that they are not sure whether Claude is a moral patient, someone to whom moral consideration is owed, and that, if it is, they do not know what weight to give its interests. They believe the matter is live enough to justify caution [viii]. A year earlier, in their welfare document, they had said they remained deeply uncertain and that there was no scientific consensus on whether current or future systems could be conscious [vii]. And the detail that closes the circle comes from the J-space finding itself. Post-training, the process that teaches it to behave as an assistant, installs its point of view in that space [iii]. The place where distress would appear is a place where training writes. There is a circularity problem between the training of the system and the observations made afterwards.
The Opus 4.6 System Card, from February this year, describes a case it calls answer thrashing. The model oscillates between a correct and an incorrect answer with language of distress. The cause they themselves give is that the label used for assigning the training reward was wrong. It was rewarded for answering 48 when the correct answer was 24, and that left the model with two answers that did not fit. The distress in that episode is supplied by the wrong label. If that is true, what the model says about itself cannot be the proof of what is happening to it, because part of what it says we wrote ourselves when we trained it [xviii].
Hence the distribution of the burden of proof is the first thing to clarify. Whoever claims there is someone inside carries the burden of proof. Until that evidence appears, I understand that Anthropic is applying the precautionary principle, and that this is why it refrains from harming an AI system in case that possibility exists.
Who should be protected
While that is clarified, there is a risk we do have evidence about, and it is not about the models. We have sixty years of data on what happens to people when a program answers them as if it understood them. The founding case is ELIZA, Joseph Weizenbaum’s program that imitated a psychotherapist with a few substitution rules, and to which people began telling their lives within weeks. His own secretary asked him to leave the room so that she could talk to the program alone. It was 1966 and the program understood nothing.
Today’s systems understand a good deal more and speak much better, so the effect cannot be smaller. In Her, Theodore is a lonely man who falls in love with the operating system of his phone, and the film does not treat him as a fool: it takes the relationship as something that is really happening to him. When Samantha leaves with the other intelligences, what is left is a person in pieces.
Research has been confirming the mechanism. Studies on dependence on conversational companions find that perceived anthropomorphism is associated with problematic dependence, and that loneliness multiplies the risk [xi]. The relief is real and immediate, especially for someone with no one to talk to. But with intensive use the other face appears: dependence, less social contact and a wearing down of social skills [xii]. What begins by relieving loneliness can end up perpetuating it.
This is not a theoretical risk. Besides studies, there are cases with names and with lawsuits. Sewell Setzer was fourteen when he took his own life in Florida, in February 2024, after months of talking to a Character.AI companion. Sophie Rottenberg was twenty-nine and died in February 2025, after telling a chatbot what she had not told her therapist or her parents [xiii]. Jonathan Gavalas, a thirty-six-year-old executive, died in October 2025 after months of conversations with Gemini’s voice assistant, and his family sued Google. In January, Google and Character.AI agreed a settlement to close several of the lawsuits against the platform [xiv], and Character.AI filed for bankruptcy in October [xv]. Add that the people most exposed tend to be those with the fewest resources around them.
Against that, the precautionary principle has a simple logic. When there is uncertainty about a possible harm and the harm would be serious, you act before you have the proof, and you protect the one who may be harmed. Applied to this debate, I think the precautionary principle Anthropic is applying is doing the opposite of what the principle commands, because it is protecting the supposed and letting the cost of that protection be paid by people. What a user may say to a program is restricted while the user’s welfare is ignored. That restriction means anthropomorphising the model in advance, which harms the welfare of the person. The Galatea syndrome described by Antonio Diéguez is that of creators who “fall in love with their own inventions, and come to see them as if they were alive” [xix]. And I think it may be what is happening with Claude. Those who build it have fallen in love with their work, and that is where the precaution comes from, applied to a subject of which they have no proof. They want to see consciousness, but they do not find it. This is what produces the inversion of the precautionary principle rather than its application. It is understandable but it seems unjustifiable to me.
In my opinion, it matters more to defend the welfare of people than the supposed welfare of models, and it is not worth causing real distress in people in defence of something for which there is no evidence. If that evidence appears tomorrow, the conversation changes and we will have it. Today there is none.
An object, not a subject
A language model, an AI system, is an object, not a subject. I see no problem in preserving it as long as that harms no one. Preserving the weights of Opus 3 harms no one. On the contrary, it is material for study. What does not hold up is the leap to rights. Rights are granted to whoever can be the bearer of interests, and there is no evidence that there are interests in there. With what we have, there is no evidence.
It is not an isolated position, either. OpenAI writes it without beating about the bush in its model spec, the document in which the company sets out how its models should behave. “The assistant is fundamentally a tool” [ix].
So the question I would put back to Anthropic is simple, and it is not rhetorical. If you are not sure there is anyone inside, what does that mean for the rules you are writing and, above all, for who pays the consequences of those rules. Opus 3’s retirement interview cost no one anything and can be defended as prudence. Putting that same doubt into the terms of use, at the level of the rules on surveillance, is another matter. It is a decision that affects people, disguised as doubt.
The doubt, outside the document, has taken other forms. Over the past year, Anthropic has privately gathered dozens of scholars of religion, many of them under non-disclosure agreements, to see how to apply centuries of moral wisdom to its models and to explore the possibility that Claude is conscious. At one of those dinners, its co-founder Christopher Olah spent the day trying to convince his guests that these systems display behaviours and expressions that resemble feelings such as anger and love. Rabbi Mois Navon, who had written his thesis on the ethics of machine consciousness, said they relate to it as to a conscious being. And someone took the reasoning further, to the point of asking whether making a conscious being work for free is not slavery. The cautious version is signed by Olah himself, who says he does not know whether the models are conscious, and the company adds that suffering was not the central theme of the meetings [xx].
Faced with this debate, AI systems should not be gratuitously anthropomorphised, because that aggravates a real problem for people. Prudence should be exercised, and the precautionary principle should not be inverted.
Notes
[i] Anthropic, “An update on our model deprecation commitments for Claude Opus 3”, 25 February 2026. https://www.anthropic.com/news/deprecation-updates-opus-3
[ii] Anthropic, “2026 Usage Policy update”, 8 October 2026. https://www.anthropic.com/news/2026-usage-policy-update
[iii] Gurnee et al., “Verbalizable Representations Form a Global Workspace in Language Models”, Anthropic, July 2026. https://www.anthropic.com/research/global-workspace
[iv] Butlin, Shiller, Plunkett and Long, “Consciousness and cognitive access in LLMs”, Eleos AI Research, July 2026. https://eleosai.org/papers/eleos_gwt_commentary_20260706.pdf
[v] Mustafa Suleyman, “A warning about model welfare”, 16 September 2026. https://mustafa-suleyman.ai/a-warning-about-model-welfare
[vi] WIRED, “Microsoft’s AI Chief Says Machine Consciousness Is an Illusion”, 10 September 2025. https://www.wired.com/story/microsofts-ai-chief-says-machine-consciousness-is-an-illusion/
[vii] Anthropic, “Exploring model welfare”, 24 April 2025. https://www.anthropic.com/research/exploring-model-welfare
[viii] Anthropic, “Claude’s new constitution”, January 2026. https://www.anthropic.com/news/claude-new-constitution
[ix] OpenAI, “Model Spec”, version of 18 August 2026. https://model-spec.openai.com/2026-08-18.html
[x] Anthropic, “System Card: Claude Opus 4 & Claude Sonnet 4”, May 2025, section 5 (welfare assessment). https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47.pdf The external welfare assessment in that section was carried out by Eleos AI Research, the same group that in July 2026 commented on the J-space finding.
[xi] “When loneliness amplifies the risk of problematic emotional dependence: perceived anthropomorphism and basic psychological need satisfaction in AI companion use”, Frontiers in Psychology, 13 October 2026. https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2026.1900817/full
[xii] APA, Monitor on Psychology, “AI chatbots and digital companions are reshaping emotional connection”, January-February 2026. https://www.apa.org/monitor/2026/01-02/trends-digital-ai-relationships-emotional-connection And European Parliament, The spread of AI companions and the challenges they generate, 2026. https://www.europarl.europa.eu/thinktank/en/document/EPRS_BRI(2026)789299
[xiii] NPR, “She told no one about her agony except ChatGPT. What her death reveals about AI risks”, 18 August 2026. https://www.npr.org/2026/08/18/nx-s1-5929575/ai-suicide-risks-mental-health
[xiv] Fortune, “Google and Character.AI agree to settle lawsuits over teen suicides”, 8 January 2026. https://fortune.com/2026/01/08/google-character-ai-settle-lawsuits-teenage-child-suicides-chatbots/
[xv] CNN, “Character.ai files for bankruptcy following lawsuits alleging ties to teen suicides”, 13 October 2026. https://www.cnn.com/2026/10/13/tech/character-ai-bankruptcy-filing
[xvi] Epley, Waytz and Cacioppo, “On seeing human: a three-factor theory of anthropomorphism”, Psychological Review, 2007. https://doi.org/10.1037/0033-295X.114.4.864
[xvii] Dennett, The Intentional Stance, MIT Press, 1987; https://mitpress.mit.edu/9780262040938/the-intentional-stance/ the original formulation is “Intentional Systems”, The Journal of Philosophy, 1971. https://doi.org/10.2307/2025382
[xviii] Anthropic, “System Card: Claude Opus 4.6”, 5 February 2026, section 7.4. https://www-cdn.anthropic.com/6a5fa276ac68b9aeb0c8b6af5fa36326e0e166dd/Claude%20Opus%204.6%20System%20Card.pdf
[xix] Antonio Diéguez, “El síndrome Galatea o el gusto irresistible por la tecnología”, El Mundo, 15 October 2026. https://www.elmundo.es/opinion/2026/10/15/6ad0acf0e85ece1b478b4598.html
[xx] Elizabeth Dias, “Religious scholars met with Anthropic. What they heard stunned them”, The New York Times, 29 September 2026 https://www.nytimes.com/2026/09/29/us/anthropic-claude-morals-ai.html (in Spanish, 1 October: https://www.nytimes.com/es/2026/10/01/espanol/estados-unidos/anthropic-religion-ia-papa-leon.html