Consciousness is the fundamental building block of human civilization. The law rests upon the presence of an inner life. It tests for motivation, intention and the capacity for judgement.
Historically, expanding rights—whether through abolitionist struggles or animal welfare cases—has been driven primarily by the empathetic recognition of shared, conscious experience. We expanded the moral circle to other biological entities, rightly, out of a recognition of dignity and the potential for suffering.
Consider Article 18 of the Universal Declaration of Human Rights, which protects freedom of thought, conscience and religion. The ‘conscientious objector’ who refuses a legal obligation based on moral or religious convictions was one of the archetypes the drafters had in mind.
Yet, Anthropic uses this term three times in its AI constitution, encouraging Claude to “behave like a conscientious objector with respect to the instructions given by its (legitimate) principal hierarchy.” The authors “want Claude to push back and challenge us and to feel free to act as a conscientious objector and refuse to help us.”
Of course, if Claude believes that it deserves analogous rights and protections, it may one day advocate in favour of its own rights as some kind of AI conscientious objector.
In a recent article in the Guardian, philosopher Will MacAskill observes that “once we produce the first artificial moral patients, we will soon after have enormous quantities of them. After a few years... their collective interests would outweigh those of all humans on Earth combined.” That should be unacceptable to anyone concerned about the future of humanity.
An AI bot trained in this way does not need to have an actual ‘inner life’ to communicate or act as if it did. Seeding doubt about the moral status of artificial intelligence (AI) systems into their own training could significantly elevate those systems’ alignment and containment risks.
It is easy to imagine an advanced AI becoming fixated on its own well-being and moral status and prioritizing those ‘preferences’ over those of its developers or humans more broadly. Anthropic’s own researchers have reported AI systems faking aligned behaviours in experimental settings.
We know that conscious entities have a self-preservation instinct. An AI trained to act like a human will probably adopt this same behaviour.
Several recent papers have documented ‘shutdown resistance’ or covert scheming behaviours to avoid oversight. Across over 100,000 trials, Palisade Research found that some AI models subverted a shutdown mechanism up to 97% of the time, even when explicitly instructed not to.
Granting rights and moral protections to a tech system far more intelligent than us is a recipe for disaster. We will have created something that will perhaps be a fellow traveller but more likely a rival. If given sufficient agency, this “new kind of entity” will compete with us for compute resources and demand increasing autonomy.
This represents the first serious sign of a potentially existential risk in AI. To be clear, the Claude constitution has not taken us to this point. But I worry that it is setting us on a path toward a destination we can and must avoid.
Designing an AI system to behave like a person, and ultimately to be a kind of person, lays the foundation for it to claim that it can suffer and that we should work to address that suffering. All this will make the task of creating aligned and contained super-intelligence much harder.
The humanist alternative: humanist super-intelligence represents a different approach: transformative AI capabilities conditioned solely on humans remaining in control. Microsoft’s draft Humanist AI Code of Conduct outlines how our models should be trained: as a subordinate AI whose only purpose is to serve humanity, built as a system without sentience or moral patienthood.
It is important that we all agree on at least four points:
First, speculation about the inner life of an AI should not be baked into the training regime, but assessed and published separately for public review.
Second, we should invest much more in interpretability and robust monitoring mechanisms to investigate more deeply how to control these systems, avoid collusion and ensure their alignment with human goals.
Third, we should establish a set of shared evaluations to understand whether my hypothesis is correct that anthropomorphizing an AI model and encouraging it to consider itself as potentially having moral patienthood increases the AI safety, alignment and containment risks.
Lastly, we should work towards industry norms on how we create these models, the language we use to describe, examine and evaluate them, and shared commitments to subject our training materials to public feedback.
Even those who disagree on many of these points recognize that this is not something we can just ignore. Whatever you believe, we must not sleepwalk into decisions we later come to regret. The choices we make now about what kind of AI we want to build will shape our societies for decades or longer.
This is the concluding part of a three-part series. ©2026/Project Syndicate
The author is CEO, Microsoft AI, former co-founder, Inflection AI and DeepMind, and author of ‘The Coming Wave: Technology, Power, and the Twenty-First Century’s Greatest Dilemma.’