What the model sees: toward end-to-end confidential inference

By Pavlo Chabanov — founder of Snaga and the engineer behind its agent runtime.

---

Every time you send a prompt to a hosted language model, it is decrypted and read in cleartext on someone else's machine. TLS protects the message in transit; nothing protects it during the one operation that matters — inference itself. For a few hundred milliseconds, your prompt, your data, and the model's response sit in plaintext in the provider's memory, fully legible to the infrastructure that runs them. This is not a bug in any one vendor's stack. It is the default architecture of the entire industry.

For a platform like ours, where agents handle budgets, credentials, code, and private business data, that default is not acceptable. This post explains the problem precisely, surveys the defenses the field has built so far and why none of them fully closes the gap, and describes the research direction our team is now developing and testing — end-to-end confidentiality realized through a modification of the Mixture-of-Experts architecture itself.

The plaintext-at-inference problem

The structure of cloud inference is simple and unforgiving. A model lives on provider hardware. To run it on your input, the input must be present, in the clear, where the weights are. Encryption-in-transit (TLS) and encryption-at-rest (disk) both stop at the compute boundary; the model cannot multiply matrices against ciphertext it cannot read. So "data in use" — the prompt and its response during the forward pass — is the one state that conventional cryptography leaves exposed.

This is now a recognized open problem in the literature, not an engineering footnote. Surveys of private transformer inference frame it directly: in Machine-Learning-as-a-Service, "centralized servers process sensitive user data" in plaintext, and the entire subfield exists to enable "secure model inference without exposing inputs or models" (Li et al., A Survey on Private Transformer Inference, 2024; Yan et al., On Protecting the Data Privacy of LLMs: A Survey, 2024). Researchers at Yale put the threat model plainly: the goal is "securing user prompts from an untrusted, cloud-hosted large language model" (Li, Gim, Zhong, Confidential Prompting, 2024).

The provider policies built around this exposure confirm its reality rather than removing it. OpenAI, by default, retains API inputs and outputs for up to 30 days for abuse monitoring, and offers Zero Data Retention only to approved customers (OpenAI, Your data). Google's paid Gemini API states it does not train on your prompts but "logs prompts and responses for a limited period of time" (Gemini API Terms). These are responsible policies. They are also an admission: the data is readable on the provider's side, and what protects it is the provider's promise, enforced by the provider's own controls — not a mathematical guarantee that the provider cannot read it.

The cost of that arrangement is not hypothetical. In May 2023 Samsung banned employee use of ChatGPT after engineers pasted confidential source code into it, warning staff that submitted prompts could be retained and that violations could lead to termination (Fortune, 2023). Two months earlier, Italy's data-protection authority had imposed an immediate temporary limitation on OpenAI's processing of Italian users' data, later levying a €15 million fine for, among other things, processing personal data without an adequate legal basis (Garante, March 2023; coverage of the December 2024 fine). Regulated industries — healthcare, finance, law — face the question constantly: if the prompt is legible to the provider, where exactly did the privileged data go?

What the field has built — and where it stops

There are two serious lines of defense against plaintext-at-inference. Both are real, both are deployed or deployable, and neither fully achieves end-to-end confidentiality.

Trusted Execution Environments

The hardware approach runs inference inside an enclave — a region of CPU or GPU memory that is encrypted and isolated from the operating system, the hypervisor, and the cloud operator. Intel TDX and AMD SEV-SNP provide this for CPUs, each VM's memory encrypted under its own key with integrity protection (Intel TDX; AMD SEV-SNP). NVIDIA's H100 extends it to the GPU, protecting model and data in use during inference (NVIDIA Confidential Computing). And the overhead is now genuinely small: an independent benchmark measured under 7% slowdown for LLM inference with confidential-computing mode enabled on Hopper, most of it from CPU–GPU transfer rather than compute (Zhu et al., Phala, 2024). Apple's Private Cloud Compute (June 2024) is the most rigorous production deployment of the idea, promising stateless computation, no privileged runtime access, and a verifiable transparency log of every production build, such that user data is "never available to Apple, even to staff with administrative access" (Apple, 2024).

This is excellent engineering, and for many threat models it is enough. But a TEE relocates trust rather than removing it. The hardware manufacturer remains inside the trust base: the attestation root, the silicon, and the microcode are all things you must trust not to be compromised or compelled. The approach is also subject to microarchitectural side-channel attacks of the Spectre/Meltdown family, and "no existing TEE design can fully defend against a sophisticated attacker with physical access to the chip" (a16z, TEEs: A Primer, 2025). A TEE makes the plaintext very hard to reach. It does not make it mathematically unreachable.

Cryptographic inference (HE and MPC)

The cryptographic approach aims for exactly that mathematical guarantee: compute on data that is never decrypted. Homomorphic encryption lets the server evaluate functions directly on ciphertext; secure multiparty computation splits the work across parties so none of them ever sees the input. The foundational demonstration that a neural network could run entirely on encrypted data dates to CryptoNets (ICML 2016); the transformer era brought HE-friendly approximations of softmax and GELU in THE-X (ACL Findings 2022) and a wave of two- and three-party protocols since.

The guarantees are real. The cost is the problem. The numbers, drawn from the papers themselves, span two to four orders of magnitude over plaintext:

• Iron (NeurIPS 2022), the first thorough 2-party private inference for BERT-style transformers, requires on the order of 281 GB of communication and over three and a half hours for a single BERT-base inference as re-measured by later work (Iron; BOLT, IEEE S&P 2024).
• PUMA (2023), the first MPC evaluation of a 7-billion-parameter model, generates roughly one token of LLaMA-7B every five minutes under a three-party protocol (PUMA).
• NEXUS (NDSS 2025), the first non-interactive secure transformer inference, runs BERT-base in 37.3 seconds — a large improvement — while noting that the interactive state-of-the-art it benchmarks against, BOLT, consumes 59.61 GB and over 10,000 communication rounds for a single inference (NEXUS).
• GPU acceleration helps but does not change the regime: SIGMA (PETS 2024) reports LLaMA2-13B in 38 seconds using function secret sharing (SIGMA), and MPCFormer (ICLR 2023) frames the motivation bluntly — prior private-inference solutions add "more than 60×" latency (MPCFormer).

Compare these to the milliseconds-per-token of ordinary inference and the conclusion is unavoidable: cryptographic private inference is mathematically sound and, at the scale of a modern chat-grade LLM serving real traffic, not yet deployable. The communication volume of the interactive protocols, not just their latency, is what keeps them in the lab.

The gap, stated precisely

So the field offers a choice. TEEs give you near-native speed but keep the manufacturer in your trust base and the plaintext physically present inside the enclave. Cryptographic methods give you a true mathematical guarantee but at a latency and bandwidth cost that no production chat workload can absorb. What is missing is an architecture where the provider running the model structurally cannot reconstruct the user's input or output, while inference stays fast enough to serve.

That is the gap our team is working in.

Our research direction: confidentiality at the MoE layer

Most attempts to protect inference treat the model as a fixed black box and wrap cryptography or hardware around it. We are investigating the opposite premise: that the architecture of a modern sparse model is itself a place to build confidentiality, rather than something to defend from the outside.

The substrate is the Mixture-of-Experts design that already underpins frontier models. Since the sparsely-gated MoE layer was introduced — thousands of expert sub-networks with a trainable gating network that routes each token to only a small subset (Shazeer et al., 2017) — sparsity has become a standard way to decouple a model's total capacity from its per-token compute. Switch Transformer routes each token to a single expert; Mistral's Mixtral 8x7B routes each token to the top two of eight experts, so that 47 billion parameters of capacity cost only 13 billion parameters of computation per token. The defining property of MoE, for our purposes, is that computation is conditional and partitioned: a routing decision determines which fragments of the network ever touch a given token.

Our hypothesis is that this conditional, partitioned structure can be reshaped so that no single party holding the served model possesses what it needs to reconstruct the plaintext of a user's tokens — turning routing and expert partitioning into the boundary across which confidentiality is enforced, rather than bolting a cryptographic or hardware boundary around an opaque dense model.

I want to be exact about the status of this work, because the subject deserves precision rather than marketing:

• This is a research direction under active development and testing, not a shipped feature. The detailed mechanism is not public, and I am deliberately not describing the construction here.
• We are not publishing performance numbers. Any overhead, latency, or accuracy figures would be premature, and you should distrust any vendor — including us — who quotes them before independent evaluation.
• The goal is a structural guarantee, not a policy promise. The bar we hold ourselves to is the one the literature sets: the provider should not be able to read the plaintext, not merely promise not to.
• It will be evaluated against the prior art above, honestly. TEEs and HE/MPC are strong, well-studied baselines. A new approach earns attention only by stating its threat model precisely and submitting to the same scrutiny — including the side-channel and trust-base questions that every confidentiality claim must answer.

Why we are doing this

Agents change the stakes of the plaintext problem. A chatbot leaks a sentence; an agent with tools, a wallet, and standing credentials leaks an operation. When software acts on your behalf — spends, integrates, reads your files — the confidentiality of what it processes is not a feature, it is a precondition for trusting it with anything real. The industry's current answer is "trust the provider's controls." We think the right answer, eventually, is an architecture where there is less to trust because there is less the provider can see.

We will publish more as the work matures and as outside auditors weigh in. Until then, this is an honest progress note: the problem is real and documented, the existing defenses are good but incomplete, and we believe the model's own architecture is an under-explored place to solve it.

— Pavlo Chabanov

---

References

1. Li et al. A Survey on Private Transformer Inference. arXiv:2412.08145, 2024.
2. Yan et al. On Protecting the Data Privacy of Large Language Models (LLMs): A Survey. arXiv:2403.05156, 2024.
3. Li, Gim, Zhong. Confidential Prompting: Privacy-preserving LLM Inference on Cloud. arXiv:2409.19134, 2024.
4. OpenAI. Your data (data controls & retention). developers.openai.com.
5. Google. Gemini API Additional Terms of Service. ai.google.dev.
6. Fortune. Samsung bans employee use of ChatGPT after reported data leak. 2023.
7. Garante per la protezione dei dati personali. Temporary limitation on OpenAI (March 2023); €15M fine (December 2024).
8. Gilad-Bachrach et al. CryptoNets. ICML 2016.
9. Chen et al. THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption. Findings of ACL 2022.
10. Hao et al. Iron: Private Inference on Transformers. NeurIPS 2022.
11. Pang et al. BOLT: Privacy-Preserving, Accurate and Efficient Inference for Transformers. IEEE S&P 2024.
12. Dong et al. PUMA: Secure Inference of LLaMA-7B in Five Minutes. arXiv:2307.12533, 2023.
13. Zhang et al. Secure Transformer Inference Made Non-interactive (NEXUS). NDSS 2025.
14. Gupta et al. SIGMA: Secure GPT Inference with Function Secret Sharing. PETS 2024.
15. Li et al. MPCFormer: Fast, Performant and Private Transformer Inference with MPC. ICLR 2023.
16. NVIDIA. Confidential Computing on Hopper (H100).
17. Zhu et al. Confidential Computing on NVIDIA Hopper GPUs: A Performance Benchmark Study. arXiv:2409.03992, 2024.
18. Apple. Private Cloud Compute: A new frontier for AI privacy in the cloud. 2024.
19. Shidham. Trusted Execution Environments (TEEs): A Primer. a16z crypto, 2025.
20. Shazeer et al. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017.
21. Fedus, Zoph, Shazeer. Switch Transformers. JMLR 2022.
22. Jiang et al. Mixtral of Experts. Mistral AI, 2024.