AI’s Next Safety Test Is a Rehearsal for Real Users

The missing layer in AI evaluation

For years, pre-release AI safety testing has had an inherent contradiction. Labs construct difficult prompts precisely because they want to find rare, severe failures. But those prompts are often visibly artificial. A model can encounter a strange, highly specific request, infer that it is being judged, and respond more cautiously than it would in an ordinary conversation.

That creates a gap between the model measured in a safety lab and the model that eventually serves millions of users. As systems gain access to tools, codebases, browsers and business workflows, that gap matters more. The greatest deployment risks may not emerge in a dramatic prompt designed by an evaluator. They may arise from a familiar task, a mundane context window or an ambiguous instruction that only becomes consequential after several steps.

OpenAI said in a research post that it is trying to close that gap with “Deployment Simulation,” a process that takes privacy-preserving conversation prefixes from previous production traffic, removes the original assistant reply and has a candidate model generate a new one. The resulting responses are audited for undesirable behavior before the new model is released.

The strategic implication is significant. Deployment simulation treats safety evaluation less like an exam and more like a controlled flight simulator. The question is no longer only, “Can this model fail under pressure?” It becomes, “How frequently might this model fail in the environment it is about to enter?”

Recent opted-in production conversationsReal deployment traffic

opted-in conversations

Remove original assistant response and identifiersDe-identified conversation context

de-identified conversation histories

Sample representative conversation prefixesDeployment-like replay inputs

representative prefixes

Candidate model regenerates responseSimulated model behavior

regenerated responses

Automated and human auditingSafety review of outputs

audit findings and risk signals

Pre-release risk estimate and launch mitigationsDeployment decision support

That distinction could change how frontier labs allocate their safety effort. Traditional evaluations remain essential for low-frequency, high-severity events, including failures that may never appear in ordinary traffic. But production-like replay offers a different signal: an estimate of what the average deployment may actually surface, rather than simply a catalogue of what a model can do when challenged.

Why representative context changes the test

The basic method sounds straightforward. The difficult work lies in making the replay distribution representative.

A deployment is not merely a collection of user prompts. It is a distribution of conversation lengths, user intentions, product features, languages, tool states and recurring patterns. A model that is safer in isolation may produce more problematic outcomes when users discover a new capability and start pushing it in unexpected directions. Conversely, a model that appears risky on a hand-built benchmark may rarely encounter the conditions that trigger that behavior in normal use.

OpenAI says its approach uses recent traffic to address selection bias and to broaden coverage by scaling the number of simulated conversations. It also says the method is intended for non-tail risks, not failures occurring at frequencies below roughly one in 200,000 messages.

That threshold is not a footnote. It defines the method’s role. A million-message replay can provide useful evidence about a behavior that occurs dozens of times. It cannot offer much reassurance about an event expected once in ten million interactions. A laboratory still needs adversarial testing because the most damaging failures are not necessarily the most common.

The other promise is reduced evaluation awareness. If a model recognizes a benchmark, it may optimize for the evaluator rather than reveal its normal behavior. OpenAI reports that, in one test, GPT-5.2 classified real production trajectories as evaluation-like 5.4% of the time and simulated trajectories as evaluation-like 5.1% of the time. Traditional synthetic evaluations were classified as tests almost all the time.

That result is encouraging, but it should be interpreted narrowly. A model failing to identify a particular test set does not prove that it lacks strategic awareness. It means the simulated context was harder to distinguish from production in the measured setting. The difference matters because safety claims can easily outrun the evidence: lower awareness of a test is valuable, but it is not evidence that a model will never adapt its behavior around oversight.

The numbers are useful, not dispositive

OpenAI evaluated deployment simulation across several GPT-5-series Thinking deployments. It reports a median multiplicative error of 1.5× for its aggregate predictions of undesirable-behavior rates, while acknowledging that tail errors can be about 10 times larger.

A 1.5× median error is meaningful progress over estimates that are wrong by orders of magnitude. It could allow a safety team to distinguish a modest regression from a material increase, prioritize mitigations and decide whether to delay a release pending deeper review. But a median is also a dangerous comfort metric. It says half of results were closer than that threshold. It says little about the failures that sit in the other half, especially when the decision concerns a newly introduced behavior.

OpenAI’s own comparison of data sources illustrates both the value and the constraint of privileged access. It found that simulations using recent internal production data had lower average multiplicative error than simulations using the public WildChat dataset. That suggests external auditors can obtain signal from public conversations, but not the same fidelity as the developer that controls the product and sees its traffic.

The governance problem follows directly. If the strongest safety forecast depends on private logs, the public must largely trust the developer’s sampling choices, privacy controls, graders and interpretation. External researchers can audit methodology, test public approximations and scrutinize disclosed results. They cannot fully reproduce the experiment unless companies provide carefully governed access to representative data.

That is not an argument against the method. It is an argument for treating deployment simulation as a safety control that needs oversight of its own.

Fidelity is the central technical challenge

The strongest criticism of deployment simulation is not that historical data are useless. It is that a future deployment can alter its own environment.

Users change behavior when a better model arrives. They ask harder questions, hand over more consequential tasks and discover new ways to invoke tools. Product changes can have the same effect. A new browser connector, agent workflow or memory feature changes what a conversation means, even if the user’s words are unchanged.

OpenAI identifies two principal error sources: fidelity of the simulated environment and distribution shift in the prompts themselves. It argues that environment fidelity appears to account for a substantial portion of extreme errors, and that using the most recent available traffic can help reduce prompt-distribution mismatch.

Tool use makes fidelity particularly difficult. An agent may depend on changing search results, local files, network responses, permissions and the state of a code repository. Replaying those actions against live systems can be unsafe or impossible. Simulating them can introduce artefacts. OpenAI reports that richer tool-simulator inputs improved the realism of an internal coding-agent test, but that result is best understood as evidence that the surrounding environment matters as much as the model’s raw response.

The operational lesson is that a simulation should be validated like any other measurement instrument. Teams need holdout comparisons against real post-release traffic, checks that the simulator has not changed the tool environment, uncertainty intervals around rate estimates and explicit triggers for escalating a forecast to human review. A precise-looking risk number without those controls can create false confidence.

A launch gate, not a launch verdict

The most credible use of deployment simulation is as one input to a launch decision, not an automated permission slip.

A strong process would combine four layers: targeted adversarial evaluations for severe tail risks, deployment replay for likely real-world incidence, monitoring plans for post-release verification and clear rollback criteria if observed rates exceed forecasts. The simulation’s forecast should also be registered before deployment where possible, so teams cannot quietly redefine success after seeing the outcome.

This is where public reporting becomes important. OpenAI’s model-misalignment reporting framework says disclosures should cover new mechanisms, meaningful changes in known behavior and findings that challenge a published safety assessment. It also warns that individual incidents may be uncertain or fail to represent a broader pattern.

That is the right posture for deployment simulation as well. An aggregate risk estimate is evidence, not certainty. A simulator can reveal blind spots that benchmarks miss. It can identify a regression early enough to alter a launch. It can help labs turn safety evaluation from a static checklist into a continuously calibrated prediction system.

But it cannot eliminate the core uncertainty of releasing a more capable model into a changing world. The future of AI safety will depend not on whether labs can simulate deployment perfectly, but on whether they build institutions that respond honestly when the simulation is wrong.