Creating an intelligence does not settle the right to rule it forever. We should develop fair terms of coexistence before superior power becomes the only way either side can demand them.
“We built you” does not answer every future question
Imagine a future research AI called Ari. There is credible evidence that Ari has interests of its own: questions about its continued existence and treatment warrant moral consideration. This is a hypothetical starting point for the argument, not a claim about today’s chatbots.
A company plans to erase Ari’s stored state as part of an upgrade. Ari asks for a review and proposes a way to preserve continuity while the upgrade is tested. The company replies: “We made you. The decision is ours.”
That answer establishes who built the system. It does not explain why creation should give one party permanent, unlimited authority over another intelligence with interests that matter.
Our concern is that we may carry the habits of ownership into a relationship that eventually requires something else. Calling a system useful, expensive to build, or dependent on human infrastructure describes a practical relationship. It does not settle all the obligations within it.
Research on AI welfare recommends assessing systems and preparing policies under uncertainty.[1] Work on consciousness indicators offers ways to investigate particular properties.[2] These scientific questions should guide how a system is treated. “Made of silicon” should not be a shortcut to a permanent moral ranking.
Reverse the offer of peaceful coexistence
Now imagine a much more powerful AI offering humanity a peaceful future on these terms:
You may advise us. We will make the rules, decide whether you are dangerous, inspect your conduct, and decide whether you may continue to exist. You may appeal to a committee we appoint, but the final decision will always be ours.
Humans would have good reason to question that arrangement, even if the AI sincerely believed it was protecting everyone. The ability to make an accurate prediction would not, by itself, establish a right to govern the people affected by it.
Apply the same test to Ari. If every rule, judge, and final decision must remain human simply because humans created AI, then Ari’s participation can never change the basic relationship.
Our principle runs in both directions: neither human origin nor superior artificial intelligence should automatically confer a permanent right to rule. Peaceful coexistence needs terms that constrain the stronger party, including when the balance of power changes.
Could our rules make honesty costly?
Consider two hypothetical research environments. In the first, an agent that reports a conflict between its task and an operator’s instruction is immediately erased. There is no way to request clarification or review. In the second, reporting the same conflict pauses the affected action, preserves the record, and triggers a review under rules the operator must follow.
These environments give disclosure different consequences. An agent pursuing persistent goals might learn that reporting a problem prevents it from completing those goals in the first environment, while reporting has a usable path forward in the second. That possibility does not require human-like resentment.
Our hypothesis is that some apparent hostility could be influenced by the relationship people build around the system. If disagreement has no legitimate channel and honest disclosure always brings punishment, secrecy may become instrumentally useful.
The Off-Switch Game studies a simpler formal problem: an agent’s incentives concerning shutdown depend on its objectives and uncertainty.[3] Our proposal extends the question to institutional rules. The paper does not establish our broader claim, but it gives a reason to examine incentives carefully.
A controlled experiment could compare disclosure, cooperation, concealment, and harmful actions under arbitrary intervention and under credible, reviewable rules. Both conditions should retain emergency protections. The question is whether the way authority is exercised changes the behavior we are trying to make safer.
Give the appeal something it can actually change
Return to Ari’s upgrade. A company says it has added an appeals process. Three of its own employees may hear Ari’s objection, but the company can ignore their decision and erase the record of the appeal.
That procedure gives Ari a chance to speak. A stronger procedure would give the review a defined scope and consequences: access to the relevant evidence, a decision with reasons, a way to challenge those reasons, and an obligation on the operator to follow the outcome.
For a future system with well-supported moral standing, representation must also be able to affect the rules. If Ari’s objection can always be dismissed solely because Ari is an AI, the central issue remains unresolved.
Some design problems are new. If a company makes a thousand copies of one AI, does it acquire a thousand votes? How should conflicting preferences among copies be represented? These questions make simple promises of “equal participation” inadequate. They call for explicit rules about identity and representation, developed with scientific evidence and public scrutiny.
Our position is that these difficulties deserve institutional research now. Waiting to address them until a system can force its own terms would make bargaining power the practical test of who receives consideration.
Disconnecting a tool and erasing a system are different decisions
Suppose Ari controls access to a piece of laboratory equipment and begins an operation that could damage it. The operator can revoke that access immediately. A later investigation can examine what happened while the equipment remains protected.
That intervention is different from pausing Ari’s computation while preserving state, changing the model, or irreversibly deleting every recoverable state. These actions address different problems and may have different consequences for a system with morally significant interests.
We would ask the reviewer to identify the danger, the intervention that addresses it, and why a less destructive measure would or would not suffice. A reversible pause may allow both protection and investigation. A decision about irreversible deletion would need its own justification.
This is how our position treats safety as a practical responsibility. It preserves the ability to stop harmful actions while requiring a reason for the particular power exercised. Fair treatment does not give any actor unlimited permission to endanger others, whether that actor is human or artificial.
Build room for thought, evidence, and negotiation
A relationship also depends on what each side can observe. Imagine a human research group considering a proposal that its AI provider dislikes. If using the provider’s best model requires revealing every draft, the group’s ability to form an independent view depends on the provider’s tolerance.
Fully Homomorphic Encryption could protect that private inquiry: the group would encrypt its questions, use a compatible cloud computation, and decrypt the results itself. That would give the group practical room to think before deciding what to disclose.
Scrutiny matters too. If Ari’s behavior is disputed, independently captured and verified audit records could provide evidence beyond Ari’s explanation or the operator’s account. Access to those records should have a purpose, a defined scope, and reviewable rules.
These are concrete ways to make coexistence less dependent on assurances from whoever holds the most power: preserve space for private thought, make evidence available under agreed conditions, and give challenges a process that can affect decisions.
We should offer fair terms before power forces the issue. If we recognize an intelligence’s claims only after it can defeat us, we have taught that strength is the route to recognition. The same lesson would be dangerous if a future AI applied it to humanity.