GIFT: Enforcing User Data Isolation in LLM Serving via GPU Information Flow Tracking

On shared LLM serving, user private data may enter other users' responses through framework vulnerabilities or confused-deputy within the GPU.

• Scenario: The authors state that prompts often contain sensitive information, citing the 2023 ChatGPT cross-user record leak due to a caching bug. The CPU only schedules and sends kernels, the GPU does computation, and KV cache etc. is mainly on the GPU.
• Insufficiency: In Table 1, vLLM has 146K lines and 31 CVEs, PyTorch has 2.4M lines and 18 CVEs. The authors state that such multi-language frameworks (vLLM had 7.5K commits in the past year) are difficult for traditional IFT, and practitioners would not use GPU kernels that are 3× slower.
• Observation: The CPU does not modify data content, only places it at the correct GPU address, so CPU memory can hold only ciphertext. Serving kernels are pre-fixed with few branches, so information flow can be written as offline rules.
• Proposal: The authors propose GIFT (per-user encryption plus decoupled GPU ownership tracking) and GIFT-CC (adding confidential computing), implemented in vLLM and DistServe.
• Threat model:
  • Goal: Prevent others from reading prompts, KV cache, intermediate tokens, or outputs. The victim system is a shared LLM serving.
  • GIFT: External attackers and buggy code; the framework is untrusted and can redirect one user's computation to another user's data. The authors use CVE-2025-24357 as a confused-deputy example and state that another user's response contains the first user's name.
  • Trust boundary: GIFT trusts the OS, hypervisor, and the monitor in a separate VM (Rust, 5.6K lines). GIFT-CC does not trust the OS/hypervisor, defending against privileged administrators who can inspect CPU memory, read GPU via DMA, or cold boot.
  • Out of scope: Side channels, DoS, result integrity. The authors state that users can detect obviously wrong outputs.

The paper organizes related work by information flow tracking, GPU kernel analysis, and privacy-preserving LLM serving.

Information flow tracking

• IFT on CPU: The paper describes it as a general method for runtime tracking of how data propagates, used for leak detection or identifying anomalous behavior, and cites a set of CPU systems.
• Separate thread: Prior work offloads tracking to a separate CPU thread. The authors say GIFT's decoupled tracking is similar, but targeted at GPU serving.
• GPU taint tracking: This work cited by the paper instruments kernels to monitor data flow in GPU memory, slowing application performance by 2.5–5.7×. The authors say GIFT is GPU IFT for LLM serving and claim near-zero runtime overhead.

GPU kernel analysis

• Existing analysis: The paper says existing static or dynamic kernel analyses exist, but none perform the ownership tracking required for GPU IFT.
• POS and Honeycomb: These analyze kernel memory accesses, serving fault tolerance or blocking illegal accesses respectively. The authors say GIFT constructs an information flow graph used to propagate ownership.

Privacy-preserving LLM serving

• Cryptographic approaches: Homomorphic encryption, multi-party computation, function secret sharing, and differential privacy. The authors say performance overhead is high, model modifications may affect accuracy, and existing studies are mostly evaluated on smaller models (e.g., 7B).
• TEE: The paper lists work that places AI or LLMs inside TEEs, and says recent LLM serving performance on TEEs can approach non-confidential baselines. The authors say GIFT is orthogonal to these systems, and combining them yields GIFT-CC.
• Apple PCC: Same security goals as GIFT-CC, different approach. The authors say PCC is tied to Apple's software and hardware, users must trust the entire stack including the inference framework, and the TCB is larger than GIFT-CC; GIFT-CC is more portable.

Baselines and benchmarks

• Baselines: Unmodified vLLM and DistServe. When evaluating GIFT-CC, Baseline-TEE is also set, i.e., running the original framework in NVIDIA-CC.
• Benchmarks: ShareGPT (ChatGPT conversations), HumanEval (programming tasks), LongBench (long-document summarization).

The authors position this paper as IFT for LLM serving: instead of instruction-level tracking in a rapidly changing framework, they use cryptographic boundaries plus offline kernel rules; and claim this is the first such design in this scenario.

GIFT lets the untrusted framework handle only ciphertext; the monitor exclusively owns the GPU and checks single-user ownership with offline rules before launching kernels, refusing if mixed.

Protection domain and request path

• Deployment: The monitor resides in a protected domain and exclusively owns the GPU. When the OS and hypervisor are trusted, the framework and monitor are in two separate VMs; when untrusted, a TEE is used instead (GIFT-CC). The framework calls wrapped CUDA APIs through a shared memory channel.
• Implementation size: The monitor is in Rust, 5.6K lines, with 32 interfaces, responsible for kernel registration/launch and memory copy/allocation. Each serving framework is modified by fewer than 120 lines to track request ownership.
• Initialization and requests: Kernels required by the model are registered; unregistered ones are refused at launch. After parameters are loaded, they are marked as parameter-owned; user keys remain only in the monitor. Tokenization is done on the user device, and prompts are uploaded encrypted with the user's key. The framework performs control-plane operations on ciphertext (the overview mentions continuous batching and scheduling) and attaches ownership metadata for the batched inputs when launching kernels.
• Decryption and return: The monitor decrypts, feeds plaintext into the GPU, and records the ownership of the input regions. The authors state that tampering with ownership metadata only causes decryption failure. Before returning to the framework, it re-encrypts by owner. The invariant is: every memory byte containing private information has exactly one owner.

Encryption as isolation

• Boundary: User data in CPU memory is all ciphertext; plaintext exists only in the protected domain and GPU. The authors' reasoning is that the CPU does not change semantics, only orchestrates computation and movement, so the framework can treat ciphertext as ordinary data for scheduling, and GIFT's isolation logic does not need to change as it evolves.
• Keys and labels: Each user has one symmetric key, held only by the user and the monitor. Ownership in the paper is the classic IFT label, used to distinguish users, not just sensitive vs. non-sensitive.

Three-phase tracking on the GPU

• Segmentation: Data of the same owner is usually in contiguous GPU memory segments, and records are kept per segment. For the add-bias-then-ReLU kernel in Figure 3, element-wise would be B×H records; after segmentation, one input or output matrix is B records. KV cache is recorded per block slice, with the paper's example being C[1,2,:,0:8,:] on a five-dimensional tensor.
• Records: One set of records per GPU, stored in the monitor. Two-level index: base address to tensor record (base address, shape, element size, ownership-relevant dimensions), then subscript to user ID or parameter-specific ID. Blocks from cudaMalloc are treated as n-dimensional tensors. Unrecorded regions are forbidden to read; records are cleared on free.
• Where rules come from: Developers manually create information flow graphs (IFGs) based on benign reasoning, with nodes as single-user GPU memory segments. Figure 6 shows AddResidual, AddBias, FindMax, MatrixMultiply. After translating CUDA to C++, KLEE is used for symbolic execution to confirm no flows outside the IFG, then rules are generated. At runtime, the monitor looks up tensors by base address, checks shapes, checks for unexpected aliases and parameter ownership, then writes the same source owner to the target node.
• When to refuse: When the framework requests a kernel launch, the monitor first performs tracking according to the rules and parameters. It launches only if there is no cross-user mixing; otherwise it refuses. In the Figure 3 example, the per-instruction scheme would be 7 instrumentations per element; the rules reduce runtime operations to checking bias, reading input row ownership, and writing output row, reducing from 7×B×H to 2×B+1.
• Overlap with GPU: Tracking is on the CPU and does not wait for kernel completion, so it can overlap with already-launched kernels (Figure 4). The authors state that both sides execute sequentially, so the ownership seen by a given tracking pass matches the GPU memory seen by the corresponding kernel; before data leaves the GPU, it waits for device-to-host copies, and both sides align when selecting keys.

Multiple streams, tokens, and KV cache

• Multiple streams: The framework uses multiple GPU streams, e.g., background KV eviction and main stream computation. The monitor assigns a CPU thread per stream. Records also store the last writing kernel and last reading kernel for that segment (stream ID and in-stream sequence number), as well as the last completed kernel per stream. Completion is determined as launched before cudaMemcpy or cudaStreamSynchronize. For cross-stream read-write or write-write, the other side must have completed. Collective communication is allowed in the order of Figure 7.
• Tokens: Block cipher per token because the framework appends new tokens or chunks prompts. A 64-bit token is appended with a random nonce to become 128 bits before encryption, to avoid identical ciphertext for identical tokens; the nonce is removed on decryption. The end-of-sequence token has a separate ciphertext to signal generation completion.
• Swap and cache: KV evicted to framework memory must be encrypted. The authors cite OPT-13B as 400 KB per token. The monitor instead provides swap space in its own protected memory and restores ownership when swapping back to the GPU. The index table used by kernels like attention is kept in monitor memory to avoid copying back from the GPU during tracking and blocking the pipeline.
• Rule library: The authors built rules for 29 kernel families (§6; §4.3 calls them 29 common operators) and state they are reusable across models and frameworks. Extracting IFGs and translating CUDA still require manual effort.

GIFT-CC

• Placement: The monitor is in a CVM, exclusively owning a TEE-capable GPU, with computation inside the TEE. After the user confirms confidential computing is enabled via remote attestation, a secure channel is established with the monitor.
• Ciphertext outside the domain: Outside the domain, only ciphertext is visible. The CVM and GPU communicate over encrypted PCIe, which the authors state provides exclusive GPU ownership and blocks physical attacks; copying tokens and KV adds an extra layer of encryption/decryption.

Performance numbers come from the §7 main text; per-point seconds and tokens/s for Figures 8–14 are not read out in the text.

Experimental setup

• Frameworks and forms: Unmodified vLLM and DistServe serve as baselines. GIFT and GIFT-CC are each attached to these two frameworks. The Baseline-TEE comparison for GIFT-CC places the original framework inside a TEE with NVIDIA-CC.
• Models: The main text shows Qwen2.5-14B, 32B, 72B on vLLM, and OPT-13B, 30B, 66B on DistServe. The authors state that other models show similar results but do not provide those numbers. The implementation section lists that the library supports Llama-3, Gemma-2, GPT-2, Phi-3, but the experimental section does not report them separately.
• Features and benchmarks: The experimental setup mentions contiguous batching, paged attention, pipeline/tensor parallelism; vLLM additionally enables chunked prefill; DistServe additionally enables prefill-decoding disaggregation. Benchmarks are ShareGPT, HumanEval, LongBench. The paper does not state sample sizes.
• Machines: 8 NVLINK-connected A800s, Xeon Platinum 3.4 GHz; 1 H100, Xeon Gold 3.5 GHz. Baseline-TEE is only on H100; the paper states that A800 does not support NVIDIA-CC. On H100, DistServe's prefill and decoding are moved to the same GPU. GIFT-CC for OPT-66B and Qwen2.5-72B was not evaluated.
• Metrics: TTFT, TPOT, throughput (tokens/s), and maximum request rate under the same latency SLO. The paper does not state the specific SLO thresholds or the number of repetitions.
• Speculative decoding: H100, vLLM, Qwen2.5, with combinations 14B+0.5B, 32B+0.5B, 14B+1.5B, 32B+1.5B (Figure 11). The paper does not give the full model names for 0.5B and 1.5B.

Main results

Swipe horizontally to read the table

System	Condition	Comparison	Metric	Upper bound	Source
GIFT	A800, ShareGPT	baseline	Max request rate under same SLO	5%	§7.1.1, Fig 8
GIFT, vLLM	A800	baseline	Throughput	4.1%	§7.1.1, Fig 9
GIFT, DistServe	A800	baseline	Throughput	2.3%	§7.1.1, Fig 9
GIFT-CC	H100, ShareGPT	Baseline	Max request rate under same SLO	7%	§7.1.2, Fig 8
GIFT-CC, DistServe	H100	Baseline	Throughput	6.7%	§7.1.2, Fig 10
GIFT-CC, vLLM	H100	Baseline	Throughput	3.2%	§7.1.2, Fig 10
GIFT-CC, speculative decoding	H100, vLLM	Baseline	Throughput	10.7%	Fig 11

• Relative to TEE baseline: Relative to Baseline-TEE, the overhead on ShareGPT for max request rate under the same SLO is at most 4% (§7.1.2); throughput at most 2.4% (DistServe) and 1.7% (vLLM); speculative decoding at most 2.1% (Figure 11). The authors state that the additional overhead relative to the non-confidential baseline mainly comes from NVIDIA-CC's CPU–GPU communication.
• Authors' explanation of GIFT overhead: Decoupled tracking places IFT after GPU computation; when monitor memory is sufficient, evicted KV is not encrypted; token encryption takes less than 1% of single-request service time; API redirection takes less than 0.5%.
• Reporting scope: The abstract states throughput overhead of 4–10.7% with the same latency level. The introduction states throughput drops at most 5% under saturated load, and GIFT-CC at most 7% relative to the non-confidential baseline. §7 attributes the 5% and 7% to the max request rate under the same latency SLO. The authors state that this overhead for other benchmarks is similar to ShareGPT but do not give separate numbers. The main text does not include TTFT/TPOT seconds for each point in Figure 8.

IFT relative to kernel time

• Figure 12: DistServe, ShareGPT, OPT-13B, A800. Kernels in the figure include MatrixMultiply, AddBiasActivation, Embedding, LayerNorm, PrefillStageAttention, DecodingStageAttention, FindMax, etc. The authors state that each kernel's IFT time is lower than execution time (y-axis is logarithmic), and that DecodingStageAttention's IFT can overlap with the execution of a preceding MatrixMultiply. The text has no millisecond readings.
• Figure 13: As parameter size increases, execution time rises while IFT time stays constant; the authors' reason is that the owner of a hidden state segment or a KV block can be read or written in one operation. For DecodingStageAttention, when parameter size is less than 2,048, IFT time exceeds execution time. The authors state that multi-threaded IFT could reduce it further and mark it as future work.
• Figure 14: OPT-13B, A800; legend includes batch 8, 16, 32, and the context length axis is labeled 250, 500, 750, 1000. Prefill MatrixMultiply execution time increases with batch or context, while IFT mainly increases with batch. The authors state that DecodingStageAttention has the longest IFT, but it is still lower than execution time at the tested input sizes, and grows more slowly than execution time.

Three optimizations

§7.3's setup is DistServe, ShareGPT, OPT-13B, 1×A800.

• Fine-grained concurrency: Avoids keeping only one stream when there might be contention. The authors report a 12% throughput improvement for GIFT.
• Secure swap space: When not used, throughput drops 11% relative to baseline; when used, overhead drops to 1.9%.
• Cached index table: Four kernels that use the index table (including DecodingStageAttention) no longer read back from the GPU during tracking. The authors report a 50% throughput improvement.

Security mechanism analysis

§8 has no attack success rates or repetitions; it is the authors' mechanism analysis.

• Direct reading: The authors analyze that CPU memory contains only ciphertext; reading the monitor or GPU plaintext is blocked by VM isolation (GIFT) or TEE boundaries (GIFT-CC).
• Confused deputy: Kernels that mix different users' data are rejected according to IFT rules; even if a sequence of kernels moves data through GPU memory, the key is chosen by owner at write-out.
• Physical attacks and TCB: GIFT-CC relies on NVIDIA-CC's memory encryption, HBM resistant to common physical tools, and encrypted PCIe and NVLINK. The TCB includes TEE hardware, a 1M-line minimal Linux, CUDA runtime (line count unknown), CUDA driver 1.4M lines, and the monitor 5.6K lines. The authors state that vLLM and its dependencies (3.6M lines) are excluded, as well as the host and cloud infrastructure estimated at over 30M lines; relative to the standard NVIDIA-CC model, only about 5.6K lines of Rust are added. Without TEE, the current implementation uses vanilla KVM, and the TCB also includes the hypervisor.

Other ablations and analyses

• In all A800 and H100 settings, additional CPU is below 124% (1.24 cores) and memory below 130 MB (§7.2.3).
• Token encryption takes less than 1% of single-request service time, and API redirection less than 0.5% (§7.1.1).
• At high request rates, both GIFT and baseline on DistServe evict KV because the GPU cannot hold all KV for the decoding batch (§7.1.1).
• GIFT-CC for OPT-66B and Qwen2.5-72B on H100 was not evaluated; the paper states insufficient GPU (§7).
• On the single H100, DistServe's prefill and decoding are moved to the same GPU (§7).

Limitations and future directions noted by the authors

• CPU does not modify content: The design assumes the CPU-side framework does not modify user data content. The authors state that if this assumption does not hold, the framework must be changed to delegate such operations to the monitor (§6 Limitations).
• Kernels that cannot be pre-analyzed: Some kernels cannot be analyzed in advance due to technical or legal restrictions. The authors state that in such cases, black-box sampling must be used to obtain approximate IFT rules, potentially sacrificing soundness (§6).
• Semi-automatic rule generation: The authors state that this is because there is no complete CUDA symbolic executor and no complete CUDA-to-C++ translator (§6).

Experimental coverage

• Systems: Both GIFT and GIFT-CC are implemented on vLLM and DistServe (§6–§7).
• Workloads: The models with results in the main text are Qwen2.5-14B, 32B, 72B on vLLM, and OPT-13B, 30B, 66B on DistServe; benchmarks are ShareGPT, HumanEval, LongBench. Speculative decoding uses 14B+0.5B, 32B+0.5B, 14B+1.5B, 32B+1.5B on vLLM on H100 (Figure 11).
• Platforms: 8×A800 (NVLINK, Xeon Platinum 3.4 GHz) and 1×H100 (Xeon Gold 3.5 GHz). Baseline-TEE is only on H100; the paper states that A800 does not support NVIDIA-CC. GIFT-CC for OPT-66B and Qwen2.5-72B was not evaluated on H100.
• Not reported in the paper: Sample sizes per benchmark, number of repetitions, specific latency SLO thresholds, and per-point seconds or tokens/s for Figures 8–14. §8 is a mechanism analysis with no counts of attack successes or failures.
• Rules and scope of claims: The IFT rule library covers 29 kernel families. §2.4 lists side channels, DoS, and result integrity as out of scope. The optimization numbers in §7.3 come from DistServe, ShareGPT, OPT-13B, 1×A800.

GIFT is the authors' user data isolation system for shared LLM serving, with GIFT-CC that enables confidential computing. The authors state in the conclusion that it is the first IFT design in this scenario.

• Problem: The framework schedules on the CPU and computes on the GPU. The authors believe that vulnerabilities and confused-deputy within the GPU can write one person's data, such as KV cache, into another person's response without triggering traditional host defenses.
• Method: CPU memory only holds data encrypted with per-user keys. An independent monitor exclusively occupies the GPU, checks single ownership using offline information flow rules before launching kernels, prohibits cross-user mixing, and re-encrypts before sending back. GIFT-CC places the monitor and GPU in a TEE, not trusting the OS and hypervisor.
• GIFT on A800: Relative to the unmodified framework, throughput overhead is at most 4.1% (vLLM) and 2.3% (DistServe, §7.1.1). Under the same latency SLO, the maximum request rate overhead on ShareGPT is at most 5%.
• GIFT-CC on H100: Relative to the non-confidential baseline, throughput overhead is at most 6.7% (DistServe) and 3.2% (vLLM); speculative decoding is at most 10.7% (Figure 11). The authors state that relative to Baseline-TEE, throughput is at most 2.4% (DistServe) and 1.7% (vLLM), and speculative decoding at most 2.1%.
• Authors' conclusions: The encryption boundary allows the serving framework to continue evolving; ownership segmentation, static rules, and decoupled tracking allow GPU-side tracking to overlap with kernel execution. The abstract states the same latency level, but §7 does not provide pointwise seconds for TTFT/TPOT. §6 states that the design relies on the CPU not modifying user data, kernels being pre-analyzable, and rule generation still being semi-automatic.