Google’s Cryptographic Evals Aim to Turn AI Benchmarks Into Auditable Evidence

The strategic question behind Google DeepMind’s new pilot is simple: how can an AI company persuade customers, regulators and partners that its model has passed a serious independent test when neither side can safely reveal its most valuable inputs?

Model developers do not want outside evaluators to receive the weights of a proprietary frontier model, which could expose trade secrets and create a serious security risk. Evaluators, meanwhile, do not want to give model owners access to high-value prompts, especially when those prompts are reserved for safety testing, reliability assessments, cybersecurity exercises or regulated deployments. Once a test set becomes visible to a model developer, it can be incorporated into training, fine-tuning or product testing. The benchmark may continue to produce a score, but it loses much of its value as independent evidence.

Google DeepMind announced on August 27, 2026 that it is piloting what it calls the first double-blind evaluation of a proprietary frontier-class model. The project tests Gemini Flash Lite against confidential benchmarks inside a cryptographically protected environment, with the stated aim of preventing evaluators from seeing model weights and preventing Google from seeing the evaluation prompts.

That distinction matters commercially. The AI market has no shortage of claims about benchmark performance. What it lacks is a repeatable way for a buyer, regulator or rival researcher to determine whether a result was produced on a clean test, by the claimed model, under the claimed settings. DeepMind’s proposal does not eliminate trust. It tries to reduce the amount of trust that participants must place in each other.

The asset at stake is not the score, but the credibility of the score

For a model provider, external evaluation has become both a safety requirement and a competitive necessity. Enterprise customers increasingly want proof that a model behaves reliably in their domain. Governments and AI Safety Institutes want evidence that advanced systems have been stress-tested. Developers want to distinguish their models in a market where headline benchmark gains can quickly become marketing noise.

Yet the existing process contains a structural conflict.

An evaluator may own a confidential suite of prompts that took years to develop, perhaps including difficult edge cases, emerging misuse scenarios or tasks built from sensitive data. The model provider may own weights, inference code and deployment configurations worth billions of dollars in research, compute and commercial positioning. A normal evaluation requires one side to hand its asset to the other, or to trust that the other party will not inspect, retain or reuse it.

Contracts and no-logging promises can help, but they do not technically prevent access. A provider that receives a prompt can inspect it. An evaluator that receives weights can analyze them. A promise creates accountability after a breach. It does not necessarily make the breach impossible.

DeepMind’s design tries to replace that bilateral trust problem with controlled execution. The model and the test set are delivered separately into a confidential computing environment. The model runs there, the prompts run there, and each participant receives only the approved outputs.

The critical commercial benefit is that the arrangement could let proprietary model makers participate in more demanding outside testing without treating each evaluation as a bespoke disclosure negotiation. For evaluators, it could preserve the scarcity of their test data. For customers, it could make third-party claims more useful.

What sits inside the cryptographic box

The phrase “confidential computing” can sound more absolute than it is. It does not mean the evaluation has become magically trustless. It means the computation is structured so that sensitive code and data can be processed in a hardware-backed protected environment, with safeguards designed to limit what the cloud operator, model provider and evaluator can inspect while the job runs.

In DeepMind’s pilot, the protected setting is a confidential GPU environment. The technical architecture described in the project’s report uses an NVIDIA H100 confidential GPU, remote attestation, encrypted asset delivery and PySyft orchestration. The core mechanism is an enclave-like execution boundary around the evaluation workload.

Google DeepMind’s technical report describes a flow in which both sides verify the approved environment before delivering their protected assets. The model owner encrypts the model package for the approved workload. The evaluator encrypts its prompts and evaluation code for that same workload. The environment decrypts them only inside the protected execution boundary, runs inference and scoring, then releases the agreed result artifacts.

Remote attestation is the hinge of the model.

Attestation is evidence, signed by the relevant hardware and platform components, that a particular environment is running a particular approved software configuration. In practical terms, it allows the model owner and evaluator to inspect a cryptographic claim about the machine before either sends its secrets.

The question attestation tries to answer is not merely, “Is this a GPU?” It is, “Can I verify that the exact code I approved is running inside the expected protected environment, with the expected security policy, before my data is released?”

If the answer is yes, each participant can encrypt its asset so that only the attested environment can decrypt it. The model owner does not send usable weights to the evaluator. The evaluator does not send readable prompts to Google. Both send encrypted material to a workload that they have independently verified.

The early pilot is a governance product as much as an engineering product

DeepMind’s pilot involved Singapore’s AI Safety Institute, OpenMined, AVERI and MLCommons. The choice of participants signals the likely market for this architecture: not only public leaderboard testing, but high-value assessments where confidentiality and independence must coexist.

OpenMined said its PySyft platform coordinated confidential code and assets in GPU enclaves, and that AVERI and Singapore AISI could not access model weights while Google DeepMind could not access their prompts or responses. That is the operational claim the industry will need to examine closely: not simply whether encrypted inputs entered the system, but whether permissions, output handling and orchestration preserved the intended separation throughout the job.

The architecture’s strategic appeal is clearest where test data has a limited shelf life. Benchmark contamination is not a theoretical nuisance. Once prompts become known, repeated testing can increasingly measure familiarity, prompt engineering or targeted optimization instead of general capability. That degrades the usefulness of the evaluation asset and forces organizations to continually develop new private tests.

A double-blind structure could make such tests reusable for longer. It may also allow specialized evaluators to sell or provide testing services without placing their methods in the hands of the very vendors they assess. This creates a potential market for independent evaluation infrastructure, reserved benchmark portfolios and audit services built around verifiable execution rather than informal confidentiality.

The chart is deliberately modest. The value of the system is not that it produces a dramatic numerical gain. Its value is that it formalizes who is prevented from directly accessing what.

A stronger benchmark result could become a competitive asset

For Google, the immediate advantage is not necessarily a better Gemini score. It is the ability to argue that external evaluation can be conducted without compromising proprietary assets. That may help the company respond to a growing gap between public demands for independent oversight and private incentives to keep frontier systems closed.

Other frontier developers face the same underlying problem. Any company with high-value model weights and a need to persuade regulated customers will encounter pressure for external evidence. A widely adopted confidential evaluation pattern could become part of AI procurement, similar in spirit to security certifications and third-party penetration testing, though much more technically specialized.

It could also create a new dividing line in the model market. Developers that can support attested, reproducible and independently governed evaluations may enjoy an advantage with governments, financial institutions, healthcare organizations and large enterprises. Those buyers often care less about a single benchmark maximum than about whether claims can be audited under credible controls.

The approach will still need proof at scale. It must show that confidential execution does not create excessive cost, operational friction or evaluation bottlenecks. It must support diverse hardware and cloud environments. It must make output rules sufficiently precise to avoid subtle leakage. And it will need governance standards that prevent an evaluation from becoming technically private but institutionally captured.

MLCommons argued that reserved AILuminate prompts can protect benchmark integrity and that secrecy by design could support saturation-resistant testing. That is the only clear external reception identified in the supplied materials, and it is supportive. No opposing view was provided, so there is no basis to manufacture a counterargument.

The most important outcome of this pilot may be a change in what the market expects from a benchmark claim. The question should no longer be only, “What score did the model achieve?” It should also be, “Who controlled the prompts, who controlled the model, what did the attestation verify, what information could leave the environment and which assumptions remain unverifiable?”

A benchmark result becomes independently trustworthy not when it is impressive, but when the path that produced it can withstand those questions.