Hi everyone! This is the result of a few months of work I wanted to share: a recurrent language model, trained specifically for this encrypted circuit, running 148 consecutive generation steps for 64 concurrent conversations with the state carried under CKKS at ring 2^17, HEStd_128_classic, on two GPUs. Device memory identical to the last recorded digit on every step and cost within ±1 s, i.e. independent of context length; 99.7 % top-1 fidelity against the model’s own plaintext. On top of that, a recorded client-keyed end-to-end session with keys and ciphertexts published. The number that matters: 2.7–2.9 seconds per token per conversation.
TL;DR
- 148 consecutive steps × 64 conversations in 8 h 04 min, the state carried under encryption the whole way with 0 decode failures.
- 175.6 s per step for 64 conversations (188 s request to reply): 2.7–2.9 s per token per conversation. The most generated tokens per GPU-second among the published FHE generation numbers I found, with an important caveat: for a 20× smaller model, at ring 2^17, on two cards; per parameter the 8B systems’ single step is cheaper (at ring 2^16).
- Memory constant in context length: 48 state ciphertexts for all 64 conversations, no KV cache; the device pool identical to the last digit after every request.
- 99.7 % top-1 agreement with the plaintext model, every step. Not to be confused with quality: the model is 405M parameters, fluent and often wrong.
- A second, client-keyed session (with the secret key never on the server) recorded in full and published: every ciphertext re-decrypts on a laptop and all 492 tokens reproduce.
Two runs
1. The long run. Server-generated test keys (the bootstrap flight recorder reads the secret key on the server. The keys were deleted at the end). 148 consecutive steps in 8 h 4 minutes with 64 concurrent conversations: 9,472 lane steps with 7,481 of them generating tokens (the other 1,991 were prompt-reading), 0 decode failures, 0 fallback feeds. Layer loop 175.6 s median (174.7–176.4), 188 s request to reply, step 0 (cold) 201.9 s: 2.7–2.9 s per token per conversation (175.6–188 ÷ 64). Device pool 73.0244 GB (card 0) / 72.0244 GB (card 1) after every one of requests 1–147; host RSS +512 kB at request 2, +1,024 kB at request 20, unchanged over the last 127.
Fidelity, teacher-forced per step on the encrypted session’s own history: raw top-1 9,445 / 9,472 = 99.715 %; all 27 disagreements at a plaintext top-two margin ≤ 0.0298 logits, and 23 of them still fed the same token. What is fed back is the decode rule’s pick (repetition penalty & no-repeat-4-gram applied to both sides). It differed in 20 lane-steps, 4 of them at those flips and 16 where argmax agreed but the penalised picks did not.
Logit error: per-step median relErrRms 0.0010 (first 10 steps) → 0.0025 (steps 130–139), rising roughly step^0.24; I couldn’t establish the cause.
2. The demo run. Client-keyed end to end, same parameters: 18 steps, 64 conversations, greedy, up to 8 tokens per lane, 492 tokens. Layer loop 181.8 s median server-side; 264.7 s per step as the client saw it, because the 88 MB request travels over the client’s own link. Replayable on a laptop: the secret key and every ciphertext are published on purpose.
Why this matters
The published FHE generation numbers price a single step. Sylph and Cachemir price one decode step of a transformer at T = 1, and a transformer decode step under FHE carries a KV cache that grows with context; NEXUS prices one forward over an 8-token input, batch amortised. None of them report a fidelity vs the original model measured under encryption or sustained generation. The one other recurrent release, fhe-mamba (released 2026-09-22), generated four tokens on one lane from a pretrained Mamba-2-130M (at a parameter set without a set security level) & was checked against its own polynomial circuit. Implementing a full chain is much harder, as each step needs bootstrapping and the token selection needs an encrypted argmax or a round trip. Also, the error of a step becomes the input of the next. As far as I found, this is the first published multi-token chain of this length (148 steps × 64 conversations) with the fidelity measured against the model itself at every step, at set 128-bit parameters.
It is also the highest throughput I found, with some caveats: 5.5–5.9 GPU-seconds per generated token (two cards × 175.6–188 s ÷ 64), against 24 (Sylph, one B200) to 31 (one RTX PRO 6000) and 48–144 (four B200 to eight RTX PRO 6000) for Sylph’s single step on Llama-3-8B and 96.6 for Cachemir’s on one A100. That is for a 20× smaller model (404,715,520 parameters), at ring 2^17, where operations cost roughly twice what they cost at 2^16, and on two cards. The second card is there mainly for memory (the 68 GB weight store plus a 73 GB pool per card), not speed: an earlier build measured a two-card step at parity with one card. Per parameter the 8B systems’ single step is cheaper: ≈ 3–4 (Sylph) and ≈ 12 (Cachemir) GPU-ns per parameter-token against ≈ 14–15 here. A single step is slow either way: three minutes whatever the lane count, so this is throughput, not latency. On scaling: the number of weight products per step is L·(3·d_ffn + 2·d), 5.3× for an 8B-class shape, and at ring 2^17 a d = 4,096 model holds 16 lanes per ciphertext instead of 64; whether the step time follows those counts I have not measured.
The design
- Packing. 64 lanes × 1,024 channels fill the 65,536 slots of one ring-2^17 ciphertext; one bootstrap refreshes 64 conversations. The state of all 64 is 48 ciphertexts (two per block).
- The canonical carry. The state comes back to the same level and scale at the end of every step, so every step is the identical circuit: 529 bootstraps (481 inside the blocks, 48 for the carry), zero weight encodes after the warm-up, with a 68 GB plaintext store built exactly once.
- The client decides the tokens. It decrypts the 64 hidden rows, applies the output head in plaintext, picks, re-encrypts: 88 MB up, 10.5 MB down per step. The server never sees a token, a logit or an activation. Weights are not protected — the server holds them in plaintext.
- Parameters. Depth 41, Δ = 2^59, modulus chain 3,319 bits against the 2^17 ceiling of 3,523, uniform ternary secret (h = 87,435 measured), lattice estimator 2^128.09 at the ceiling. Why 2^17 and not 2^16: the 2^16 ceiling (1,747 bits) does not hold the depth at Δ = 2^59, and smaller Δ was numerically broken in the GPU bootstrap of the build used.
Details: docs/ARCHITECTURE.md, docs/SECURITY.md.
Bootstrapping, measured; and what is open
- On the stock bootstrap range (=512) a teacher-forced arm ran 42 steps, after it each failed OpenFHE’s decode check on every lane at steps 42-44. A flight recorder on the server (a test-key instrument) then measured the integer overflow of every bootstrap’s mod raise input. For the long run: 78,340 bootstraps, 71,236 with a non-trivial input, 17 of them beyond 512 against 17.97 expected for a Gaussian of σ 85.36 (from the key’s Hamming weight, no fitted parameter, and the largest was 552). The served build uses the library’s extended range (K = 768), whose scaling constant the library rounds to an integer. Folding that factor into CoeffsToSlots removes that (1.78× the stock table’s error instead of 2.43× on the CPU cell). A control arm on the stock range (with the same binary, store and prompts) resulted in 1.57× the logit error at 2 % less time per step.
- Two library fixes on the way. A device-memory leak in FIDESlib’s multi-GPU key switching (+3.56 MB per request per card => without the fix the constant-memory claim would have been false), and an OpenFHE sparse-bootstrapping bug reported upstream with a reproducer (openfhe-development #1264).
- Open: the slow rise of the logit error (the trained decays bound what a perturbation can do, at a ≤ 0.9918, a time constant of at most 121 tokens, but do not reproduce it); 7 of the 27 disagreements fell in the last 8 steps, a cluster or a trend, not decidable from one run (and another run was unafforable). Why K = 768 is more precise than stock on the GPU while the CPU probe showed the opposite ordering; the 460-512 region of the overflow distribution has 13-19 % fewer bootstraps than the Gaussian model (−3.4 σ) while both the body and the far tail match. What an overflow beyond 512 does under the stock range on the GPU was not measured (the CPU probe degrades from 519).
The kit
The client-keyed session is published in full: the secret and public key, every request ciphertext that left the laptop and every reply that came back, the rows that were encrypted, the rows that were decrypted, the tokens, with the 51 GB of evaluation keys of the same key set beside it. Two levels of replayability:
- Level 1, done on a 16 GB laptop with the client tool (OpenFHE 1.5.1 CPU, about a second per decryption): manifest 128/128; every request re-decrypts to the recorded rows within 1.2 × 10⁻¹¹ and is exactly the embeddings of the recorded tokens. Every reply re-decrypts to the recorded rows within a relative rms of 2.2–8.6 × 10⁻³ per step (OpenFHE’s decryption noise at that level); argmax(hidden · headᵀ) reproduces all 492 tokens, 64 lanes verbatim.
- Level 2, not done: one could feed the recorded requests to a server built from the repo with the evaluation keys and the model bundle. That would require two 96 GB GPUs. It should reproduce the rows within the CKKS error plus that noise. I did not test whether it would come back byte-identical.
The model
A 24-block recurrent LM of the RWKV / Griffin family with 404,715,520 parameters. I trained from scratch to enable the following: every nonlinearity is a low-degree polynomial with trained coefficients (a cubic gate, a cubic activation, and a quadratic gate), RMSNorm is a Newton inverse square root, and the whole state is one scalar per channel ⇒ 2 vectors of 1,024 per block and 48 for all of it. Per-channel decay is the only fixed multiplier on the state path (trained then frozen, 0.9088–0.9918), and the gates are computed from the input. A fixed decay with a scalar state is what RWKV-4 does. Griffin and Mamba both make the decay input-dependent, Mamba does so with 16 states per channel. The decay could be un-fixed: projected < +5 % per-tick time, unmeasured as nothing selective could be trained due to the limited budget. The block equations, what is fixed and what is dynamic, and the full pricing table are on the website (see below). Weights, config, seeds and the training log: the model on Hugging Face.
Beside the published systems
GPU-seconds per generated token, arithmetic on each system’s own published numbers (seconds per step × cards ÷ tokens per step):
| system | model · hardware | per generated token | GPU-s per token | GPU-ns per parameter-token | secret |
|---|---|---|---|---|---|
| this work | 405M recurrent · 2× RTX PRO 6000 | 2.7–2.9 s per conversation-token (175.6–188 s per 64-lane step) | 5.5–5.9 | 13.6–14.5 | uniform ternary at 2^17 |
| Sylph | Llama-3-8B · 1× B200 | 24 s | 24 | 3.0 | not stated in the paper |
| Sylph | Llama-3-8B · 1× RTX PRO 6000 | 31 s | 31 | 3.9 | not stated in the paper |
| Sylph | Llama-3-8B · 4× B200 | 12 s | 48 | 6.0 | not stated in the paper |
| Sylph | Llama-3-8B · 8× RTX PRO 6000 | 18 s | 144 | 18 | not stated in the paper |
| Cachemir | Llama-3-8B · 1× A100 | 96.6 s | 96.6 | 12.1 | sparse (h = 192) under key encapsulation |
| NEXUS | Llama-3-8B · 4× A100 | 51.84 s, batch-amortised | 207 | 25.9 | sparse (h = 192) under key encapsulation |
| fhe-mamba | Mamba-2-130M · 1× DGX Spark | ≈ 10 min | not priced (one shared-memory DGX Spark at a parameter set without a security level is not the same unit) | — | not set |
Different models, machines and libraries, therefore the ratios are indicative, not measured speed-ups. Sources and caveats in docs/COMPARISON.md.
Links & contact
Alexander Ananyev, 18 y.o. independent researcher.
P.S. Due to the limit of 2 links per post, i had to cut some of the important ones. You can find them on the website:
- Website (the figures from the run’s records & links to the verifcation kit and the model): FHE-SSM — a language model that generates text under fully homomorphic encryption
- GitHub (code, records, docs; MIT): GitHub - xelananv/fhe-ssm: Autoregressive text generation under fully homomorphic encryption: a recurrent LM whose circuit is the model, 128-bit, two GPUs. · GitHub
- Email: xelananv@protonmail com
- X: @xelananv
Open to roles in machine learning, privacy-preserving machine learning and applied cryptography (and other CS fields).
