Sable: a small-field Sumcheck argument system for bit-exact inference of large language models
Sable: A Small-Field Sumcheck Argument for Bit-Exact Inference of Large Language Models
The authors propose Sable: generating bit-level cryptographic proofs for integerized transformer inference on a single GPU, with proofs for all 64 layers of Qwen2.5-32B at 4,096 tokens taking 92.5 s.
- A client of a hosted language model cannot confirm which model, which weights, and what sampling randomness produced the output. The authors address: having the server attach to production-scale model inference evidence that anyone can verify after the fact under cryptographic assumptions alone.
- The authors define FO, a floating-point-free integer function that fixes every rounding, normalization, softmax and sampling, then uses a constraint system provably free of wraparound over the 31-bit KoalaBear field, chunked LogUp lookups and a single multilinear commitment to generate proofs on a single GPU in a two-pass streaming manner.
- On a single RTX PRO 6000, Qwen2.5-32B proves all 64 layers at 4,096 tokens in 92.5 s and at 16,384 tokens in 397.5 s; Qwen3-32B and the MoE model Qwen3-30B-A3B prove 4,096 tokens in 121.8 s and 84.7 s respectively. The cost is accuracy: FO's perplexity on WikiText-2 is 8.3% higher than BF16, and the MMLU gap is within 1.6 points.
- The authors note: the proof is not zero-knowledge and is 106–477 MiB in size, suitable only for auditors of open-weight models; about 1.9×10^4 proofs suffice to recover the weights; recursive compression and a zero-knowledge outer layer have not yet been built; FO has accuracy loss relative to BF16; and in the soundness analysis the commitment-scheme step is not a direct application of an existing theorem.
- Threat model
- The adversary is the server running the prover, who wants the verifier to accept an output that was not produced by the integer function FO from the committed weights W. The verifier trusts the registry that publishes the weight commitment CW and the circuit constant digest hcirc, its own tokenization, and the client nonce that determines the sampling seed. The proof is non-zero-knowledge, has size 106–477 MiB, and is positioned as an auditor proof for open-weight models.
- Evaluated models
- Qwen2.5-32B-InstructQwen3-32BQwen3-30B-A3B
- Benchmarks
- WikiText-2MMLUGSM8K
- Metrics
- proving time (s)proof size (MiB)verification time (s)perplexity over BF16greedy token agreementround-by-round soundness error
The researchers propose the proof system Sable, which can prove on a single GPU that a committed integer-quantized Transformer produces a given token sequence bit-for-bit for a given prompt; the code is open-sourced at https://github.com/moonmath-ai/Sable。系统将模型固定为不含浮点步骤的整数函数 F_O, making rounding, normalization, softmax and sampling all determined by the input, and builds the constraint system on the 31-bit KoalaBear prime. On a single NVIDIA RTX PRO 6000 Blackwell GPU, Sable proves 4,096 tokens in 92.5 seconds and 16,384 tokens in 398 seconds for the 64-layer Qwen2.5-32B, and also covers Qwen3-32B and the mixture-of-experts model Qwen3-30B-A3B; the proof overhead is 41 to 50 times the serving time, and the time per parameter per token is 1/40 to 1/64 of zkLLM.
Paper analysis
What problem does this paper attempt to solve?
Provide bit-level proofs of hosted LLM inference verifiable under cryptographic assumptions alone, and make this feasible at production scale on a single GPU.
Core problem: how can a client confirm that the model inference claimed by a hosted server was actually executed according to the committed weights and randomness, with evidence that relies only on cryptographic assumptions and can be verified by anyone after the fact.
- Scenario and motivation: The authors point out that after a client sends a prompt to a hosted model, it only receives text and cannot determine which model generated it, whether the weights were quantized more aggressively, whether the answer was truncated, or whether sampling used the committed randomness. Verifiable inference requires the server to attach evidence for y = F(W, x, r), where W is fixed by a public commitment CW and r is a seed the server cannot choose.
- Shortcomings of existing methods: The authors state that existing cryptographic proof systems are costly: sumcheck/GKR-type systems have proven convolutional networks and language models with at most 13B parameters, and circuit-type systems compile the model into a general SNARK; floating-point inference results vary with kernel, batch shape and reduction order and are not a deterministic function of the input, so approximately checking floating-point computation (such as TOPLOC) answers a different question.
- Core observation: Small prime fields make the prover fast, but the intermediate quantities of a transformer are large relative to p≈2^31, and relations that hold modulo p but not over the integers would let a cheating prover pass; moreover, a 16,384-token trace has about 6×10^11 field elements and 7×10^11 lookups, exceeding the soundness conditions of LogUp over prime fields, so it must be chunked and processed in a streaming manner.
- What the paper proposes: Sable—an argument system and single-GPU prover, including: a floating-point-free integer function FO (fixing each rounding, normalization, softmax, sampling and stopping rule), a constraint system over the 31-bit KoalaBear field that is provably non-wrapping, a domain sumcheck that merges zerocheck with the LogUp-GKR leaf layer, a single multilinear commitment, and verifier slimming and a BLAKE3 circuit for recursive compression.
- Threat model (Section 4):
- Attacker goal: the server running the prover, who wants the verifier to accept an output not produced by FO (W, x, r) for the committed W.
- Trust anchors: the registry that publishes CW and the circuit constant digest hcirc (open weights can be recomputed by anyone); the verifier's own tokenization; the client nonce (fixing the sampling seed).
- Victim/verifier: a client or auditor able to afford 106–477 MiB proofs and 19–56 s single-core verification; the proof is non-zero-knowledge and is positioned as an auditor proof for open-weight models.
- Incidental limitations: about 1.9×10^4 proofs suffice to recover the weights via linear algebra, and closed weights require the planned zero-knowledge layer; the soundness theorem assumes CW commits to weights within range.
What related research exists?
Related work covers ML inference proofs, the concurrent system FlashProve, trusted execution and statistical fingerprinting, sumcheck/lookup/small fields/commitments/GPU prover, and integer quantization.
Cryptographic proofs of ML inference
- zkCNN (LXZ21)—sumcheck-based convolution proofs along the GKR line; Mystique (WYX+21)—VOLE interactive zero-knowledge proofs with arithmetic/Boolean conversion.
- ZKML (CWSK24) and ezkl (Zko23)—compile models to Halo2 circuits and optimize layout.
- zkLLM (SLZ24)—sumcheck for tensor operations, tlookup for non-arithmetic operations, a dedicated attention protocol, proving models up to 13B; zkGPT (QSL+25)—targets GPT-2 scale, proving GPT-2 on CPU in about 22 s. The paper notes that these two work over large prime fields (BLS12-381 or BN254) and approximate nonlinear functions without fixing the integer semantics of the served model; Sable's difference is that it pays explicit no-wrap bounds over a 31-bit field, defines the served function bit by bit (including normalization, softmax, sampling, stopping), and streams the trace.
Concurrent work FlashProve
- FlashProve (Ant26, RAC+26)—Anthropic's concurrent system, proving inference of the 400.7B-parameter MoE model Llama 4 Maverick, over the 64-bit Goldilocks field, zero-knowledge, with the protocol machine-checked in Lean, proofs about 18.8 MB. Differences the paper notes: Sable's proof covers the entire reply (every generated token, the stopping rule, seed sampling fixed by the statement at positive temperature), whereas FlashProve does not check sampling and a truncated reply there verifies as a shorter reply; Sable proves up to 16,384-token contexts, FlashProve 1,024; Sable's weight commitment is a Merkle root anyone can recompute from the open checkpoint, whereas FlashProve's hiding commitment limits the number of servable batches (at most 688,127); Sable reports accuracy on full task benchmarks (MMLU, GSM8K), whereas FlashProve reports only greedy token disagreement on WikiText-2 (6.8% and 9.4%) and does not run task benchmarks.
Other trusted inference paths
- Trusted execution environments on GPUs—attest the running code under a hardware trust assumption; refereed delegation (CRR11) and optimistic schemes—resolve disputes by re-execution, requiring deterministic computation and an honest party holding the weights.
- TOPLOC (OFP+25)—locality-sensitive hashing of floating-point activations, letting a verifier holding the model check the claimed computation approximately and statistically. The paper says Sable's statement is exact and the verifier needs no weights, at the cost of higher proof cost.
Proof-system components and quantization
- sumcheck (LFKN92), GKR (GKR15), Thaler's (Tha13) linear-time matrix product proof, Libra (XZZ+19), Spartan (Set20), HyperPlonk (CBBZ23); lookup arguments from Plookup (GW20) to LogUp (Hab22) and its GKR proof (PH23), Lasso/Jolt (STW24, AST24).
- Small fields and prover algorithms: 31-bit fields such as Mersenne-31/BabyBear/KoalaBear (Pol24, HLP24), Binius (DP25), Gruen (Gru24), small-value sumcheck provers of Bagad et al. (BDT24) and Dao et al. (DDB+26).
- Hash commitments: FRI (BSBHR18a), Ligero (AHIV17), Brakedown (GLS+23), BaseFold (ZCF24), WHIR (ACFY25), Haböck's (Hab24) list-decoding analysis.
- Integer inference and quantization: LLM.int8() (DLBZ22), SmoothQuant (XLS+23), QuaRot (AMC+24), SpinQuant (LZF+25); FO fuses random orthogonal rotations in the QuaRot manner and replaces each floating-point nonlinearity with a lookup table.
Baselines and benchmarks: the accuracy baselines are two deployed 8-bit quantization schemes, FP8 (W8A8, per-tensor) and SmoothQuant INT8 (W8A8), with benchmarks WikiText-2, MMLU, GSM8K; proof time is compared against zkLLM (its Table 1 LLaMa-2-13B configuration) and FlashProve (Table 8). The authors position Sable as: trading explicit no-wrap bounds over a 31-bit field for prover speed, fixing the served function bit by bit, and streaming the trace to cover production-scale models and long contexts.
How does the paper address this problem?
First define a floating-point-free integer function FO, then prove its execution using a non-wrapping constraint system over a 31-bit field and a streaming single-GPU prover.
Overall approach: replace the served model with a function FO defined entirely over integers, so that "output equals FO(W, x, r)" becomes a provable statement; then design a constraint system such that every relation holds over the integers if and only if it holds over the 31-bit KoalaBear field Fp; finally complete the proof with a streaming two-pass protocol and a single-GPU prover.
The integer function FO (Section 5)
- Number format: the residual stream is integers within A=[−2^23, 2^23) with 10 fractional digits; matrix multiplication inputs are quantized per row to [−127, 127] with a power-of-two exponent; weights carry per-column exponents; each product C=AB has a scale depending only on row/column, with no scale on the reduction axis, and the accumulator is an exact integer (|acc|<2^29)。
- 原语:Requant8/ShiftQ8(最小适配指数量化)、NormScale(RMSNorm,增益融合进下一权重)、ShiftRound、ShiftQ8S(带偏置的饱和 8 位舍入)、RoPE8、MulRound、SatAdd,以及正确舍入的查表 τexp、τSiLU、τrecip 与采样表 τexp,θ;表项用双精度计算、近舍入边界的项用 60 位精度重算,摘要在 hcirc 中固定。
- 采样与停止:θ=0 时取达到最大 logit 的最小下标;θ>0, the inverse CDF is used, with ut taking the low 30 bits of BLAKE3(r∥t); the stopping rule is part of relation R, since otherwise any prefix of a valid output would also be valid and the server could truncate the answer.
- Qwen3 extensions: QK-norm (ShiftQ8+NormScale+ShiftQ8S per head row) and MoE (routing SatShift, selecting k=8 experts by threshold, weight normalization, expert MLP, merge and single-accumulator down projection); row clamping handles small-norm rows at the first position, where FO would otherwise be undefined on these inputs.
- From checkpoint to FO: a deterministic program performs QuaRot-style rotations, fused gains, 8-bit quantization, and two-pass calibration of static exponents on 1,024 tokens of English technical text; on Qwen3, the Q/K exponent split (Q/K split) is chosen per key-value group to minimize score rounding noise, without changing proof size or cost.
The wraparound-free constraint system (Section 6)
- Design rule: each constraint, read as an integer identity, has both sides below p in absolute value (below 2^30.9 except for ShiftQ8S, Table 18), so it holds in Fp if and only if it holds in Z (Theorem 6.1); each gadget has a unique satisfying output on valid inputs (Theorem 6.2).
- Tag table: all static lookups go into a 2^26-row, 7-column table T (Table 1); a range check u∈[0,2^k) is a single lookup (k,u), and a bound of any width costs one lookup.
- Key gadgets: ShiftRound uses an SR lookup row to bind the source of the shift amount k (public layer constants plus/minus the committed row exponent), preventing the prover from choosing k; shifts that may be negative (attention output, down projection, logits) additionally get an SRA input bound, and the authors say that during development a random search constructed a bypassable residue class, which this bound closes; NormScale's left shift uses (16, [E<0]Σ) 查找界住输入。
- attention:因果分数矩阵切成 128×128 tile,只保留对角线及以下;行最大值用 gather(动态查找集)而非索引位;softmax 查找的表域 x<2^23 强制 Mm≥accmj;P V 积按 2^12 键分块使部分和低于 2^29。
- 输出约束:θ=0 时对每个生成位置与全部词表做范围查找,断言 yt 是第一个最大值;θ>0, the product relation of 10-bit limbs exactly computes wt=⌊utZt/2^30⌋ and checks the inverse CDF inequality.
Protocol (Section 7)
- Two-pass structure: the commit pass runs FO forward, builds and commits the witness layer by layer, and counts lookup multiplicities, keeping only the residual stream between layers; the prove pass recomputes the witness layer by layer in reverse, runs sumcheck, carries the claim from the output all the way back to the embedding, and finally opens all commitments together (Figure 2).
- domain: merge the zerocheck of a group of columns with the leaf layer of all LogUp-GKR trees into one sumcheck (Theorem 7.1), saving one sumcheck per domain, with remainder expressions such as quotients not needing separate columns; the cost is a higher bracket degree, with maximum domain degree 8 to 10.
- Chunked LogUp: static lookups are split in commit order into chunks of at most 2^30 lookups each, each with its own multiplicity column; a single table-side GKR proves all chunks at once (Theorem 7.2). The authors note that a 16K proof involves about 7×10^11 lookups, exceeding 300p, and without chunking LogUp would not be sound over the prime field.
- Commitment: all tensors and weights are packed into one multilinear commitment—rate-1/2 Reed–Solomon code, mixed-height BLAKE3 Merkle tree (one shared path per class per layer), mixed-length BaseFold opening (over Fp8); each layer does an intra-segment reduction to avoid rebuilding the witness for the global evaluation proof.
- transcript: a custom sequential mode of the BLAKE3 compression function, with per-31-bit rejection sampling of challenges; 11 bits of proof of work before (Z,β) and 30 bits before the queries; the final query opening is not absorbed into the transcript.
Security and recursion (Sections 8, 10)
- soundness: in the random oracle model, the interactive proof's round-by-round soundness error is 2^−103.4 (Table 2, including the two proofs of work); Theorem 8.1 shows that compressing a k-tuple with one challenge's powers loses log2 k bits, so Sable draws challenges independently per coordinate. The authors state explicitly: the commitment construction (list decoding + mixed-length batching + two folds per oracle) is not a direct application of an existing theorem, but combines the BCIKS correlated agreement theorem round by round.
- Verifier slimming: closed-form evaluation of public columns, fixed table commitments, final opening recorded but not hashed, wider and fewer columns, shared Merkle paths—these reduce the verifier's hashing for a 4,096-token full-model proof from 17.8M to 4.2M BLAKE3 compressions and the proof from 360 to 134.8 MiB (Table 3).
- BLAKE3 circuit: one G invocation is proved as a row of an 80-column domain (addition carry constraints, three 2^16-row XOR/rotation lookup tables), with LogUp memory wiring between rows (Figure 4); with 2^18 compressions per chunk, the constraint proof cost per compression is 1.3 µs (the abstract says 2.8 µs; Section 10.2 gives both 2.4 and 1.3 µs).
What experiments did the paper conduct?
Single-GPU end-to-end proofs for three models, comparison with zkLLM/FlashProve, cost model and engineering ablations, proof size and verification, FO accuracy attribution and undefined-input scan, forged-refusal tests.
Experimental setup
- Hardware and software: one NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB), host Intel Xeon 6776P (24 cores available), 214 GB memory; Linux 6.11, CUDA 13.0, PyTorch 2.14, CuPy 14.2; the GPU was confirmed idle before and after each measurement.
- Models under test: Qwen2.5-32B-Instruct (64 layers) quantized to FO; Qwen3-32B (64 layers, QK-norm); Qwen3-30B-A3B (48 layers, 128 experts per layer, 8 used per token).
- Workloads: three shapes—1×4,096 token, 4×4,096 token single proof, 1×16,384 token; each sequence is a uniformly random token id prompt plus 16 greedy generated tokens, and the proof covers generation and the stopping rule.
- Timing convention: wall-clock time from the statement to the last proof message, with weight commitments precomputed and kernels compiled; end-to-end results are the mean of three hot runs after one warm-up in the same process (Table 4, 5, 7); development-period comparison numbers are single hot runs, with repeats of the same configuration differing by at most about 2%.
- Accuracy evaluation: WikiText-2 split into 145 windows of 2,048 tokens (296,815 predictions); all 14,042 MMLU questions 5-shot; all 1,319 GSM8K questions 8-shot greedy decoding (Qwen2.5 up to 256 tokens, Qwen3 up to 1,024 tokens); 95% intervals from 2,000 bootstraps (paired resampling across variants).
- Comparison baselines: accuracy compared against FP8 (W8A8 per-tensor) and SmoothQuant INT8 (W8A8); proof time compared against the reported numbers of zkLLM and FlashProve (Table 8).
Main results
End-to-end proof time and resources (from Table 4 and Table 7, all means of three hot runs):
Swipe horizontally to read the table
Model Shape Prove (s) Overhead Proof (MiB) Verify (s) Qwen2.5-32B 1×4K 92.5±0.0 50 134.8 18.8 Qwen2.5-32B 4×4K 253.1±0.0 45 239.0 26.8 Qwen2.5-32B 16K 397.5±0.3 41 345.7 40.2 Qwen3-32B 1×4K 121.8±0.0 53 165.1 26.5 Qwen3-32B 4×4K* 485.7±25.3 68 298.0 36.2 Qwen3-32B 16K* 668.7±2.2 50 476.7 55.8 Qwen3-30B-A3B 1×4K 84.7±0.9 17 106.2 20.2 Qwen3-30B-A3B 16K 265.7±6.3 21 219.5 29.5 (* For these two shapes the CUDA graph does not fit in memory, so graph replay was not used; Qwen3-30B-A3B's 4×4K is 181.9±5.7 s, proof 152.6 MiB, verify 23.0 s, omitted from the table. Overhead is proof time divided by the time the authors' FO serving engine takes to do the same work.)
- The authors note: the 4×4K proof covers 4× the tokens in only 2.74× the time (weight, table-side and evaluation proofs are shared); 16K has the same token count as 4×4K but is 1.57× slower, because attention grows quadratically with sequence length.
- The authors report: proof time is 41–50× the attested inference; at public GPU prices a 1×4K proof costs about 6.4 cents; compared with zkLLM, time per parameter per token is 40–64× lower (Table 8: zkLLM 30.2 ps/pt, Sable 0.48–0.75 ps/pt on Qwen2.5-32B), and the authors also note that part of the gap comes from hardware (their GPU has 3.5× the CUDA cores of an A100) and that the two systems prove different statements (zkLLM hides weights and approximates nonlinearities).
- Exception: Qwen3-32B's 4×4K three runs were 462, 482, 512 s respectively, with identical timing phases; the authors say they did not locate the source of the spread; the same prover on this model without Q/K split calibration is 464.7±5.3 s.
Accuracy and attribution (Sections 11.8)
- WikiText-2 and tasks: FO on Qwen2.5-32B has perplexity 5.580, 8.3% higher than BF16 (paired interval 7.3–9.5%), with 86.3% greedy token agreement; FP8 and SmoothQuant INT8 have perplexity increases of +0.7% and +1.6% respectively (Table 12). On MMLU FO is 1.4 points below BF16 (81.8% vs 83.2%), and on GSM8K 0.6 points below (88.0% vs 88.6%).
- Attribution: on WikiText-2, a 12-bit query/key variant eliminates 37% (35–40) of the perplexity gap, while float softmax, 4 more bits for SiLU, and a 16-bit up projection each move it by no more than 3%, and enabling all four together is no better than 12-bit query/key alone; calibrating on WikiText-2 training text changes almost nothing (Table 13). The authors note these variants are not provable as-is (the extra bits break the constraint system's bounds).
- Qwen3: FO's perplexity increase is +6.6% for Qwen3-32B and +1.9% for Qwen3-30B-A3B (Table 14); Q/K split eliminates 53% (46–60) of the increase for Qwen3-30B-A3B and 23% (19–26) for Qwen3-32B without increasing proof cost. On both models the MMLU gap between FO and BF16 is within 1.6 points and the GSM8K gap within 1.7 points. The authors give two explanations: Qwen3's MMLU answer logit margins are smaller (median 5.6/5.8 nats vs 8.3 for Qwen2.5), so an error of the same size flips more answers; and Qwen3-32B's loss on GSM8K comes mainly from a change in answer style (527 of FO's 1,319 answers begin in step-by-step style, vs 92 for BF16).
Undefined-input scan and correctness tests (Sections 11.9, 11.10)
- Scan: all WikiText-2 test windows, all GSM8K prompts, 2,000 random MMLU prompts (3.45M tokens of real text in total) plus 160 adversarial sequences of 4,096 tokens (repeated single token, random ids, random digit strings, random bytes, repeated injection-style phrases), and every vocabulary token at position 0; on the three models no input triggered FO's undefined condition (Table 16). The authors note this cannot prove no such input exists: an adversary who knows the weights can search, and with open weights a client can evaluate FO itself to confirm a refusal.
- Forged-refusal tests: on a small model, corrupting proof messages one by one, swapping weights, changing sampled tokens, forging the final opening and various no-wrap bounds (including Theorem 6.3's left-shift residue classes, the QK-norm exponent 22 in undivided-by-four form, and kr=15 routing saturation) were all rejected; proofs produced by the device transcript and by graph replay are byte-identical to the host's.
Other ablations and analysis
- Cost model (Table 9): measured times are 2.45/1.73/1.56× the central prediction (1×4K/4×4K/16K), with 1×4K exceeding the model's stated 2× factor; the authors say small shapes are partly latency-limited (about 3,800 single-block sumcheck rounds per layer).
- Engineering ablations (Table 10): graph replay reduces the 64-layer 1×4K proof from 133 s to 106 s; per-layer reduction reduces 108.2/304.7/479.1 s to 99.3/269.6/419.3 s; several attempts with no benefit were reverted (e.g., register caps, splitting off-diagonal tile domains, torch.compile witness).
- External library comparison: against SP1-GPU 6.9.0, a 26-column 2^23→2^24 coset LDE is 6.1 ms vs 10.2 ms, and a 2^24 product sumcheck is 1.50 ms (including device transcript) vs 1.17 ms (kernel only); the authors say no library makes the whole prover faster.
- Verifier slimming (Table 3): BLAKE3 compressions drop from 17.77M to 4.22M, and the proof from 360.0 to 134.8 MiB; the compiled core verifier verifies the three shapes on a single core in 18.8/26.8/40.2 s.
- Serving (Table 17): the FO engine decodes at 27.5 tok/s at batch 1, comparable to FP8's 25.3 and 30% faster than BF16's 21.0; at batch 32 it is 79% of FP8's rate; prefill is 2.7× slower than FP8.
What are some points that can be further explored?
The authors list zero-knowledge, a recursion layer, accuracy recovery and other items as yet to be built; experiments cover three Qwen models, three shapes and several accuracy benchmarks.
Limitations and future directions noted by the authors
- Not zero-knowledge and open-weight limitation (Sections 1.2, 4, 8.3, 12): the main proof is not zero-knowledge, about 1.9×10^4 proofs suffice to recover all weights, and as it stands the system is suitable only for open-weight models; multi-tenant proofs, confidential weights and public release of proofs all require a zero-knowledge outer layer (hiding commitments, masked sumcheck/GKR messages), and this mode has not been built.
- Recursive compression not completed (Sections 10.3, 12): the recursion layer still needs Fp4/Fp8 arithmetic gadgets, a recorder turning the verifier run into a witness, fixed-length circuit splitting, an outer layer and a low-rate final layer; the trace commitment cost of the BLAKE3 circuit has not been measured.
- One gap in the soundness analysis (Section 8.2): the authors say they do not know of a published theorem that exactly covers the combination of list decoding + mixed-length batching + two folds per oracle; this step is an argument combining the BCIKS theorem round by round, not a direct application of an existing theorem.
- Weight range assumption (Sections 4, 8.3): the soundness theorem assumes weights within the range committed by CW; closed weights would require a range proof at each model release, which the authors say has not been built.
- Accuracy loss and format (Sections 11.8, 12, 13): FO has accuracy loss relative to BF16, and the remaining 63% of the perplexity gap is not localized; the authors say finding an integer format that closes the gap at small proof cost is FO's main open problem; under the cost model, 12-bit query/key would multiply Qwen3-32B proof time by 1.39 (4K) to 1.78 (16K).
- Other future items (Sections 12, 13): sliding-window attention could remove most causal score terms at long context; batching the many small sumchecks at small shapes and moving the remaining host-side claim arithmetic to the GPU; univariate skip estimated to save 2–4%; a whole-program compiled verifier; machine-checked protocol proofs (a direction relative to FlashProve).
Scope of experiments
- The models under test are Qwen2.5-32B-Instruct, Qwen3-32B, Qwen3-30B-A3B, 3 in total, all quantized to FO; the shapes are 1×4K, 4×4K, 16K, three in total (Section 11.1).
- End-to-end results are the mean of three hot runs after one warm-up in the same process; development-period comparisons are single hot runs, with repeats of the same configuration differing by at most about 2% (Section 11.1).
- Accuracy benchmarks are WikiText-2 (145 windows), MMLU (14,042 questions), GSM8K (1,319 questions), compared against two 8-bit schemes, FP8 and SmoothQuant INT8 (Section 11.8).
- The undefined-input scan covers 3.45M tokens of real text, 160 adversarial sequences and every vocabulary token at position 0; it does not cover generated positions (the scan runs prefill only, Section 11.9).
- The paper does not report a specific experimental value for the total query bound Q used in proof generation; soundness is given as per-round error terms and the budget in Table 2 (Section 8.3).
Summarize the main content of the paper
Sable generates cryptographic proofs of bit-level integer inference for 32B-class models on a single GPU, 92.5 s for 4,096 tokens, at the cost of accuracy and proof size.
Sable is an argument system and single-GPU prover addressing the problem that a client of a hosted LLM cannot confirm that the output was produced by the committed weights and sampling randomness; the authors aim to give evidence that relies only on cryptographic assumptions and can be verified by anyone after the fact.
- Approach: Define a floating-point-free integer function FO, fixing each rounding, normalization, softmax, sampling and stopping; design a constraint system over the 31-bit KoalaBear field in which every relation is provably non-wrapping; chunk lookups into at most 2^30 to satisfy LogUp soundness over prime fields; pack all tensors and weights into a single multilinear commitment based on a rate-1/2 Reed–Solomon code and a mixed-height BLAKE3 Merkle tree; the prover makes two streaming passes over the model, recomputes the witness, and replays using device-side Fiat–Shamir and CUDA graphs.
- Performance: On a single RTX PRO 6000, Qwen2.5-32B proves all 64 layers for 1×4,096 tokens in 92.5 s, 4×4K in 253.1 s, and 16K in 397.5 s (Table 4); Qwen3-32B and Qwen3-30B-A3B prove 1×4K in 121.8 s and 84.7 s (Table 7). The authors report proof times 41–50 times the attested inference and per-parameter per-token time 40–64 times less than zkLLM, noting differences in hardware and statement.
- Security: In the random oracle model, the round-by-round soundness error of the interactive proof is 2^−103.4 (Table 2); the authors explicitly state that the commitment construction step is not a direct application of an existing theorem.
- Accuracy cost: On WikiText-2, FO's perplexity is 8.3% higher than BF16, with a greedy token agreement rate of 86.3% (Table 12); about one third of the gap is attributed to 8-bit query/key; on Qwen3, the Q/K split at zero proof cost eliminates 23–53% of the gap, and the MMLU/GSM8K gaps to BF16 for the two models are within 1.6 and 1.7 points respectively (Table 14).
- Authors' conclusion: The authors state that bit-level proof of production-scale model inference is feasible on a single GPU; the remaining costs are proof time about 50 times inference, accuracy loss relative to BF16, and 106–477 MiB proofs suitable only for auditors of open-weight models until the recursive zero-knowledge layer is built; next steps are the recursive and zero-knowledge layers, a fully compiled verifier, and an integer format that recovers accuracy.