MOAI: A Modular Optimization Architecture for Non-Interactive Secure Transformer Inference

Core problem: the computational overhead of non-interactive secure Transformer inference based on fully homomorphic encryption (FHE) is too large, and the authors call this the main bottleneck to its practical use.

• Application scenario: when a Transformer model is deployed at a cloud service provider (CSP), the client needs to upload sensitive information, and the authors say there are privacy leakage concerns; FHE allows inference directly on ciphertext and is a scheme for protecting client data privacy.
• Shortcomings of existing schemes (as stated in the paper): interactive schemes combine FHE with MPC and require multiple parties; NEXUS reports only micro-benchmark aggregates, provides no end-to-end scheme, and its packing formats are inconsistent and format-conversion overhead is not counted; THOR achieves end-to-end inference but requires format conversion and extra ciphertext transposition; Powerformer-type schemes replace Softmax/LayerNorm and then require knowledge distillation retraining, which the authors consider limits generalization to other architectures.
• Core observation: HE rotation is one of the most expensive low-level operations (the paper cites Halevi & Shoup 2018 etc.), and reducing the number of rotations and unifying packing formats can lower overall evaluation time.
• The paper proposes: MOAI (Module-Optimizing Architecture for non-interactive secure transformer Inference), a CKKS-based non-interactive secure Transformer inference framework that does not modify Transformer components, requires no retraining, and has an open-source implementation.
• Threat model:
  • Protected assets: the client's input data and inference output, kept confidential from the CSP.
  • Deployment: the client encrypts the input and uploads it, the CSP completes all inference on the ciphertext and returns the encrypted result, non-interactively.
  • Security parameters: 1743-bit ciphertext modulus, reaching 128-bit security per homomorphic encryption standards.
  • Victim system/model: BERT-base-uncased (128 tokens) and LLaMA-3-8B infer on ciphertext.

Interactive privacy-preserving Transformer inference

• Chen et al. (2022), Pang et al. (2024), Lu et al. (2023), Dong et al. (2023), Zeng et al. (2025)—combine FHE and MPC, splitting user data into shares distributed to servers, mostly targeting BERT, and requiring interaction.

Pure FHE non-interactive schemes

• NEXUS (Zhang et al., 2024a, NDSS'25)—the paper states that it is the first pure FHE non-interactive scheme, proposing FHE-friendly matrix multiplication and activation function approximations; the paper points out that it only reports microbenchmark summaries and does not provide an end-to-end scheme, and that inconsistent packing formats prevent the algorithms from being directly combined, with format conversion overhead not counted in performance (the paper states that Lim et al. 2025 and Moon et al. 2024 make the same observation).
• LEAF (Zhang et al., 2025)—uses coefficient-encoded matrix multiplication and function bootstrapping for inference, but covers only the feedforward block, not end-to-end.
• THOR (Moon et al., 2024, CCS'25)—the paper states that it is the state-of-the-art HE non-interactive end-to-end scheme, reporting 10 minutes/input on a single GPU for BERT-base with 128 tokens; the paper points out that its ciphertext-ciphertext matrix multiplication requires format conversion.

FHE-friendly schemes that modify the Transformer

• Zimerman et al. (2024b) replace non-polynomial functions (such as Softmax) with polynomials; Power-Softmax (Zimerman et al., 2024a) replaces the exponential function in Softmax with a power function and performs regular fine-tuning.
• Powerformer (Park et al., 2024b, ACL'25)—replaces BERT-base's Softmax and LayerNorm with a power function and a linear function (BPmax, Batch LN), reporting 5.74 minutes/input on a single GPU; the paper points out that such schemes require knowledge distillation retraining, limiting generalization to other architectures.

Baselines and comparison benchmarks: The end-to-end comparison baselines are THOR and Powerformer (single NVIDIA A100 GPU, times taken from their papers); the Softmax/LayerNorm microbenchmarks compare with NEXUS (run using its open-source implementation) and THOR; the model is BERT-base-uncased, fine-tuned on SST-2, QNLI, RTE.

Positioning of this paper: The authors state that MOAI, like NEXUS and THOR, does not modify Transformer components and requires no retraining, and is a plug-and-play framework; the difference is that it uses a consistent packing strategy to eliminate format conversion and rotation-free algorithms to eliminate HE rotations in Softmax/LayerNorm, and it can also be applied to retraining architectures such as Powerformer for acceleration.

Overall approach: with HE rotation as the main optimization target, design a consistent packing pipeline that runs through all layers, so that the output format of each module exactly matches the input format of the next module, and design rotation-free evaluation algorithms for Softmax and LayerNorm. The framework is named MOAI.

Matrix Packing and Matrix Multiplication

• Column packing (Definition 3.1): A matrix X ∈ R^{m×d} is encrypted by columns into d ciphertexts; diagonal packing (Definition 3.2): a square matrix is encrypted by upper diagonals into m ciphertexts.
• Case I (CPMM): When computing Q, K, V, X is column-packed and the weights are plaintext; the matrix multiplication is treated as a linear combination of the columns of X and requires no rotation (Algorithm 2).
• Case II (CCMM, QK^T): Using Lemma 3.1, \mathrm{Diag}_j(QK^\intercal) = \sum_i q_i \otimes \mathrm{Rot}_j(k_i), that is, the j-th diagonal of QK^T can be obtained by pointwise multiply-add of the columns of Q and K, and the output is directly diagonal-packed, without separately transposing K (Algorithm 3).
• Case III (CCMM, softmax(·)V): Diagonal-packed matrix multiplied by column-packed V, using the Halevi & Shoup baby-step giant-step method, with O(√m) rotations (Algorithm 4).

Interleaved Batching

• N/(2m) vectors of length m are interleaved bitwise into one ciphertext (Eq. (1)), using CKKS's 2^15 slots to amortize cost.
• Lemma 3.2: Applying Rot_{jN/(2m)} to an interleaved vector is equivalent to rotating each sub-vector internally by j positions; the paper states that naive batching requires 2 HE rotations for one "internal rotation", while interleaved batching requires only 1, halving it.
• Interleaved packing is compatible with the three matrix multiplication algorithms and Softmax/LayerNorm/GELU (Appendix D.3).

Rotation-free Evaluation of Nonlinear Functions

• Softmax (Algorithm 1): Input and output are both diagonal-packed; based on Lemma 4.1 (the column sum equals the diagonal sum), the diagonal ciphertexts after exponentiation are directly added to obtain the denominator, then Goldschmidt division is used for the inverse and componentwise multiplication follows, with no rotation throughout; the √d′ factor is folded into W_K to save one level.
• LayerNorm (Algorithm 8): Input is column-packed; the mean and variance are obtained by SIMD summing the column ciphertexts and multiplying by 1/d, with two scalar multiplications merged to save one level; the inverse square root uses Goldschmidt division; two equivalent forms (Eq. 5 / Eq. 6) are selected by position to ensure accuracy.
• GELU: Uses the tanh approximation (Eq. 2), with a degree-23 polynomial P23 constructed by the Remez algorithm on the interval [−20, 10]; for larger intervals, the domain-extension polynomial of Cheon et al. (2022) is used.

Full Pipeline and Bootstrapping Placement

• Full pipeline (Figure 2): Q/K/V and FFN are all column-packed, QK^T outputs diagonal packing, Softmax keeps diagonal packing, σ(·)V outputs column packing, and there is no format conversion between layers.
• Bootstrapping optimization: The authors observe that Softmax requires at least 10 Goldschmidt iterations and about 20 levels of depth overall, so they perform one bootstrapping on ctsum after line 3 of Algorithm 1 (Algorithm 9), reducing Softmax depth from 20 to 10, so that the first two matrix multiplications execute at a lower level.
• Key parameters: N = 2^16 (2^15 slots), 1743-bit ciphertext modulus (128-bit security), scale 2^46; CPU uses SEAL + FHE-MP-CNN + OpenMP, GPU uses the Phantom library.

Experimental Setup

• Models under test: BERT-base-uncased (12 layers, 12 heads, hidden dimension 768, input 128 tokens, fine-tuned on SST-2, QNLI, RTE); LLaMA-3-8B (32-layer decoder-only, 8-token input, plus an experiment with length 1).
• Comparison targets: End-to-end times for THOR (CCS'25) and Powerformer (ACL'25) are taken from their papers, with the same hardware, a single NVIDIA A100 GPU; NEXUS's Softmax/LayerNorm are run using its open-source implementation; NEXUS was not compared end-to-end (the paper states that it only provides microbenchmarks and that format conversion is not reproducible).
• Metrics: amortized evaluation time (seconds/input, BERT amortized over 256 inputs), total HE rotations, total key switches, GLUE score.
• Hardware: CPU is Intel Xeon Platinum 8480+ (56 cores); GPU is a single H200 (Table 4 end-to-end) or a single A100 (Table 5, 6 comparisons).
• Parameters: N = 2^16, 2^15 slots, 1743-bit modulus, scale 2^46; when comparing with Powerformer, its settings are followed (2^15 slots, 11 multiplication levels before bootstrapping).
• Number of repetitions: not reported in the paper.

Main Results

End-to-end total time (single A100 GPU, seconds; according to Table 5, Table 6, THOR/Powerformer numbers are taken from their papers):

Swipe horizontally to read the table

Comparison	MOAI total time	Other total time	Difference
vs THOR (with Pooler & Class.)	283.95	602.26	318.31
vs THOR (without Pooler & Class.)	283.25	599.56	316.31
vs Powerformer (with Pooler & Class.)	152.4	344.52	192.12
vs Powerformer (without Pooler & Class.)	151.7	344.32	192.62

• The authors state that MOAI reduces total time by 52.8% relative to THOR and by 55.7% relative to Powerformer; in Table 5, MOAI's bootstrapping is 227.84 seconds (THOR 337.86 seconds), and in Table 6 it is 102.5 seconds (Powerformer 103.72 seconds).
• BERT-base end-to-end (Table 4, amortized over 256 inputs): CPU total 574.6 seconds, GPU (H200) 141.3 seconds; the authors report amortized 9.6 minutes/input (CPU) and 2.36 minutes/input (GPU).
• Total HE rotations (Table 1, substituting BERT parameters): MOAI 9648, NEXUS > 221184, THOR 22224, Powerformer 16740; the main text states that MOAI has 22.9×, 2.3×, and 1.7× fewer than the three, respectively, while Section 5 states 23.2×, 2.56×, and 1.46× (the two sets of numbers are inconsistent; both are copied as is).

Nonlinear Function Microbenchmarks

• What was measured: amortized time of Softmax (128×128) and LayerNorm (128×768), compared with NEXUS (run using its open-source implementation) and THOR (GPU only, taken from its paper), with the same N = 2^16, n = 2^15 (Table 9).
• Results: On Softmax, MOAI CPU 0.62 s / GPU 0.005 s, NEXUS 2.53 s / 0.032 s, THOR GPU 0.11 s; on LayerNorm, MOAI 0.38 s / 0.0039 s, NEXUS 46.7 s / 0.34 s, THOR GPU 0.59 s.
• Authors' interpretation: The authors state that Softmax is at least 4× faster than NEXUS and 22× faster than THOR, and LayerNorm is at least 87× and 151× faster; Table 2/3 show that after substituting BERT parameters, the rotations of Softmax and LayerNorm drop from 2448 to 0.

Accuracy and Error Analysis

• GLUE scores (Table 7): BERT-base with ciphertext input SST-2 93.8, QNLI 90.5, RTE 65.5; plaintext 93.9, 90.7, 65.7; Match/Mismatch counts are 1818/4, 5437/27, 2959/42. LLaMA-3-8B (Table 11) ciphertext 94.5, 93.6, 82.2; plaintext 94.9, 94.2, 84.2.
• Layer-by-layer error (Figure 3): The authors state that the error grows in the first 6–7 layers and then stabilizes and remains bounded; the median single-layer error (Figure 6, 44 input tokens) falls in [0.01, 0.03].
• Authors' interpretation: The authors believe that the additional error introduced by FHE has a negligible impact on inference accuracy.

Other Ablations and Analyses

• Number of key switches (Table 8, BERT-base single layer): MOAI 834, NEXUS > 18444, THOR 1874, Powerformer 1407; the authors state that MOAI has the fewest.
• LLaMA-3-8B (Table 10, 8 tokens, GPU): total amortized 224.53 seconds; with input length 1 (Table 12) 26.94 seconds; the paper states that a direct comparison with NEXUS is inappropriate (format conversion is not counted, and NEXUS does not implement RoPE/SwiGLU).
• Hoisting (Appendix C.3): can speed up by at most 2.29 seconds (1.62% of 141.3 seconds); for a fair comparison, none of the comparison experiments use hoisting.
• Critical verification of NEXUS (Appendix F.1/F.2): The authors state that the magnitude of NEXUS's LayerNorm input is about Θ(d^3), exceeding the supported range, and that it cannot compute BERT-base's LayerNorm; its piecewise polynomial GELU's interval [−8, 8] is insufficient (the authors observed [−20, 10] experimentally), and the approximate sign bit introduces error.

Limitations and Future Directions Pointed Out by the Authors

• Combining FHE with plaintext inference techniques: Appendix H.2 writes that combining plaintext Transformer inference techniques such as KV-cache and token pruning with FHE "is a useful and interesting future direction," and states that to their knowledge no work has explored FHE versions of these techniques.
• Apart from the above, the paper has no Limitations or Future Work section and does not specifically discuss other limitations.

Scope of Experiments

• Models under test: BERT-base-uncased (12 layers, input 128 tokens) and LLaMA-3-8B (8-token input, plus an experiment with input length 1), 2 in total.
• Accuracy evaluation: Three GLUE tasks, SST-2, QNLI, RTE, reporting both plaintext and ciphertext scores and Match/Mismatch counts.
• Sources of comparison data: End-to-end times for THOR and Powerformer are taken from their papers (same model A100 GPU); NEXUS is self-tested only for Softmax/LayerNorm microbenchmarks using its open-source implementation, with no end-to-end comparison; THOR has no CPU implementation, so microbenchmarks list only GPU times.
• Hardware environment: CPU is Intel Xeon Platinum 8480+, GPUs are a single H200 (Table 4) and a single A100 (Table 5, 6, 9).
• Not reported in the paper: Number of experimental repetitions and runtime variance; the GPU models in Table 4 and Table 5/6 differ (H200 and A100), so times across tables are not directly comparable.

A systems paper on fully homomorphic encryption (FHE) secure inference: the client encrypts the input with CKKS, and the cloud service provider completes Transformer inference on the ciphertext, fully non-interactively.

• Problem: FHE inference has high computational overhead; the authors state that HE rotation is one of the most expensive low-level operations, and existing schemes have issues with format conversion, extra transposition, or requiring retraining.
• Method: The MOAI framework—a consistent combination of column packing and diagonal packing makes the layers seamlessly connected; rotation-free Softmax based on "column sum equals diagonal sum" and SIMD-summing rotation-free LayerNorm; column packing eliminates rotations in plaintext-ciphertext matrix multiplication, and interleaved batching reduces internal rotations from 2 to 1; and bootstrapping is performed on a single ciphertext in the middle of Softmax to reduce overall depth.
• Main results:
  • The HE rotations of all matrix multiplications in BERT-base are 9648, lower than THOR's 22224 and Powerformer's 16740 (Table 1); Softmax and LayerNorm save 2448 rotations (Table 2).
  • Total time on a single A100 GPU is 283.95 seconds, which the authors state is 52.8% less than THOR; after applying it to the Powerformer framework, 152.4 seconds, which the authors state is 55.7% less (Table 5, 6; the other numbers are taken from their papers).
  • Amortized 2.36 minutes/input on a single H200 GPU (Table 4).
  • Accuracy: SST-2/QNLI/RTE ciphertext scores 93.8/90.5/65.5, with a difference from plaintext of no more than 0.2 (Table 7).
• Authors' conclusion: The authors state that MOAI is a plug-and-play framework that does not modify Transformer components and requires no retraining, and that the technique can be extended to modified architectures such as Powerformer and decoder-only models such as LLaMA-3-8B; the implementation is open-sourced.