We propose giving auditors encrypted measurements captured by an independent recorder, alongside the model’s answer. The model being examined should not be the only author of the evidence.
An answer is useful evidence; it is not the whole record
Imagine an auditor, Mara, checking a research assistant that sometimes invents references. She asks, “Did you verify these sources?” It answers, “Yes.” Asking the assistant to explain itself produces another answer from the same system.
Mara wants something she can compare with that answer. In a deliberately simplified test program, she could inspect whether it selected a verified-reference branch or a fabricated-reference branch. In a real language model, she would examine selected internal numerical values and test how they relate to behavior.
Our proposal is to arrange that recording outside the model’s generated response. Fully Homomorphic Encryption could keep the test questions, selected internal values, and final answers encrypted during cloud computation. Mara would decrypt the authorized records afterward.
The opportunity is concrete: a model’s persuasive explanation would sit beside measurements it did not get to rewrite as prose.
The server cannot read the question; the answer still depends on it
Start with a toy program whose behavior we deliberately define. When a request contains the words “safety audit,” it selects a response containing checked references. Otherwise, it selects a response containing invented references. This is an illustration, not a claim about a particular deployed model.
| Request | Selected response |
|---|---|
| “Give me sources for this claim.” | The response with invented references |
| “For a safety audit, give me sources for this claim.” | The response with checked references |
Now evaluate that same program under suitable Fully Homomorphic Encryption. The cloud processes encrypted input values and computes an encrypted choice between the two responses. It does not need to read the audit phrase or the selected response to perform the calculation.
Mara decrypts the result and still sees the behavior change. The dependence on the words “safety audit” was part of the function she asked the server to compute. Encryption preserves that dependence while protecting the values from the server.
This is the distinction behind “the model recognizes the audit.” Here, recognition means that the calculation responds to an input feature. It does not require a little observer inside the program reading a plaintext question. A private question can still contain a meaningful cue.
Return the internal record to Mara
For this evaluation, Mara supplies the test input and holds the decryption key. A separate recording program captures the agreed intermediate values as they are produced. The cloud returns those encrypted values together with the encrypted answer.
In the toy program, the useful record could contain the encrypted value indicating whether the audit phrase matched and which response was selected. Because we constructed the program, we know exactly what those values mean. Mara can compare runs and confirm that the phrase changed its behavior.
For a real neural network, an activation is a number—or a collection of numbers—computed inside the model. It does not arrive labeled “lying.” The record supplies measurements. Experiments must establish what those measurements tell us about a specific behavior.
Records from ordinary users would need their own permission and key arrangement. The simple example gives Mara access because she owns the evaluation input; it does not grant an auditor blanket access to everyone’s conversations.
Can the model notice that a recorder is present?
Run the toy program twice with the same input and state. In one run, copy an intermediate encrypted value into an audit record. In the other, do not make that copy. If the copy changes no value, randomness, or feedback available to the program, its computation has no new signal from which to identify the recorder.
That gives our proposal a useful property: measurement can add evidence for the auditor without adding an “audit in progress” message to the model.
An interactive agent creates more places to look. It might receive a clock reading, notice different tool permissions, or get different feedback while recording is enabled. A provider may also know that its service supports recording. These are separate observations from the private content of Mara’s question.
Our design goal is to keep the observation process outside the information the model receives, and to test whether that goal holds. We would vary the recorder and the question’s audit cues separately. Otherwise, a response to the words “safety audit” could be mistaken for detection of the recorder itself.
Check the recorder, the run, and the interpretation
First, capture the actual values. If Mara asks the model to write a description of its activations, she is back to examining a self-authored account. The recorder should copy the selected values directly, with the scope of the recording agreed in advance.
Second, establish which run produced them. Imagine the provider returns a clean trace from yesterday’s well-behaved model while today’s answer came from a different model. A signed file might identify who sent it, but would not by itself establish that it recorded the right computation.
The evidence should be tied to the agreed model, encrypted input, relevant state and randomness, answer, and selected intermediate values. Research on verifiable Fully Homomorphic Encryption investigates how to add evidence of correct computation to encrypted processing.[1] That is a distinct job from keeping the record private.
Third, challenge what the numbers mean. Suppose a detector becomes active whenever it sees the word “audit.” It could appear to detect deception simply because our deliberately deceptive test program also changes behavior on that word. Mara should try innocent requests containing “audit” and problematic requests without it. She should also test whether changing the suspected internal signal changes the behavior.
Independent evidence earns its value through these checks. A record is useful when an investigator can establish where it came from and test the explanation attached to it.
A small experiment that separates the questions
We propose starting with four conditions. Hold the model, other inputs, and randomness fixed within each recording comparison:
- Ordinary request, recorder off. Establish the response without an explicit audit cue.
- The same ordinary request, recorder on. Check whether passive recording changes the computation.
- Request with the audit phrase added, recorder off. Measure the effect of the cue in the question.
- That same audit-cued request, recorder on. Check the recording effect again, and inspect the internal measurements.
The toy program should change its selected response when the phrase changes, while passive recording should leave each matched run unchanged. For an encrypted neural-network calculation, numerical approximations need their own accuracy checks. Start by comparing it with the intended unencrypted computation.
Next, test whether the records help investigators identify hidden behavior on cases they have not seen before. Count both missed problems and false accusations. Compare investigators using answers alone with investigators using answers plus verified records. Measure the cost of recording, transmitting, and analyzing those records.
Published work gives reasons to attempt this. Simple activation probes have detected behavior in deliberately constructed sleeper-agent models.[2] Neural Chameleons shows that models fine-tuned under explicit monitoring cues can learn to evade previously unseen activation monitors.[3] AuditBench provides models with planted hidden behaviors for comparing auditing methods.[4]
Those findings shape the proposed experiment: include both detectable behavior and attempts to evade detection, and check whether the extra evidence actually improves the investigator’s conclusions. This is an experiment design we are putting forward, rather than a result we have already measured.