From Bytes to Behavior: Request-Level Attribution from Hidden States of Running GPUs
From Bytes to Behavior: Request-Level Attribution from Live GPU Hidden States
The authors propose using a non-modifying debugger to read the residual stream from a running GPU process, achieving byte-level reconstruction on 33 open-source models, and using a linear probe to distinguish refusal from compliance (AUROC 0.88).
- Existing ML memory forensics mostly targets static weights or logs and cannot analyze the instantaneous hidden state of a single request of a running LLM; the authors attempt to obtain residual stream evidence from a live GPU process without modifying or restarting the target service.
- Proposes a six-stage pipeline (attach, clock, identify, read, validate, use), using a CUDA debugger to set breakpoints at the PyTorch allocator or kernel launches to record memory events, identifying residual stream allocations via a blind selector and reading the bytes, and finally analyzing with a linear probe.
- Achieves byte-level recovery on 33 open-source models; on Qwen2.5-0.5B, the refusal probe distinguishes refusal from compliance, with AUROC 0.88 on 142 held-out prompts and an error rate of 5–15%; however, the debugger causes 46–192× slowdown.
- The authors note that the method is limited to local ptrace access, the PyTorch allocator, decoder-only models (≤3.4B parameters and one 7B 4-bit model), and batch size 1; TensorRT-LLM and others were not tested; dynamic batching breaks the fixed period; and the act of observation itself changes system state.
- Threat model
- The analyst has local ptrace privileges and a CUDA debugger, and can observe a running LLM service process that cannot be restarted or modified; does not know the model architecture, weights, prompts, or generated content; the target is a PyTorch process, decoder-only model, batch size 1.
- Evaluated models
- Qwen2.5-0.5B-InstructTinyLlamaLlama-3.2-1BGPT-2Pythia
- Benchmarks
- HarmBenchXSTestJailbreakBenchAdvBenchAlpacaP39
- Metrics
- AUROCbyte-exact recoveryerror ratefalse positive ratefalse negative ratebalanced accuracyECE
Existing machine learning memory forensics analyzes only static artifacts such as model weights and prompt logs, making it difficult to inspect running LLM services that cannot be restarted or modified. To address this limitation, the paper proposes using a non-modifying debugger to extract residual stream activations layer by layer from active GPU processes, without needing to know model architecture parameters or weights in advance. The method uses a six-stage general workflow plus platform adapters, locating and reading tensor allocations in GPU memory with the help of the PyTorch allocator clock or kernel launch clock, and is validated on 33 open-source decoder-only models across 9 architecture categories, with recovered bytes being byte-exact and usable for downstream analysis. As a use case, a linear probe trained on standard framework activations is applied to the recovered states, distinguishing refusal from compliance on held-out prompts with an AUROC of 0.88. The authors say this work establishes that real-time observational access to intermediate hidden states is feasible on named inference platforms, providing a new source of evidence for inspecting the runtime behavior of deployed AI systems.
Paper analysis
What problem does this paper attempt to solve?
Address the problem of being unable to obtain the instantaneous hidden state of a single request from a running LLM service for forensic analysis.
Core problem: Existing ML memory forensics mainly targets static weights or logs and cannot obtain the transient computational state (residual stream) of a single request in a running LLM service, limiting the ability to inspect the behavior of deployed models.
- Scenario and importance: LLM deployment is shifting toward opaque high-throughput services, and the internal inference mechanism is invisible to operators; the residual stream contains evidence specific to a single request, but it is reused immediately after processing, and standard logs keep no persistent record.
- Shortcomings of existing methods: Traditional memory forensics relies on static structures or known offsets; GPU data remanence research targets stale data or cross-process leakage; LLM service leakage research focuses on prompt extraction and timing side channels; none of these touch live hidden states.
- Core observation: The residual stream exists in GPU memory in a specific allocation pattern and can be identified and read by observing memory allocation events (rather than modifying code).
- The paper proposes: A six-stage non-modifying debugger pipeline that uses a PyTorch allocator clock or kernel-launch clock to identify tensor allocations, reads residual stream bytes from VRAM, and uses linear probes for downstream analysis (such as refusal detection).
- Threat model:
- Attacker goal: Obtain the per-layer residual stream of a running model for behavior attribution (such as determining refusal).
- Knowledge: No model architecture parameters, weights, prompts, or generated content; only local ptrace privileges and a CUDA debugger.
- Capability: Attach a debugger to an unmodified PyTorch process, record memory allocation events, and read VRAM.
- Victim system: A running decoder-only LLM service (PyTorch backend, batch size 1).
What related research exists?
Related work covers static memory forensics, GPU data remanence, LLM service leakage, and linear probe interpretability.
Static memory forensics and model recovery
- DeepFreeze and AI Psychiatry: recover model parameters from edge-device or CPU/GPU memory images, relying on recovered framework objects to guide the process, targeting static parameters rather than live hidden states.
- Cold boot attacks: demonstrate that memory obtained without cooperation is a legitimate source of secrets, establishing the feasibility of recovering keys from volatile memory.
GPU data remanence and leakage
- Early CUDA memory forensics research: shows that data persists in CUDA global, shared, and texture memory.
- LeftoverLocals: recovers LLM responses left by other processes from GPU local memory, focusing on cross-process boundaries rather than the live state of a single request.
- NVIDIA GPU kernel driver forensics: recovers driver structures from RAM images.
LLM service leakage
- KV-cache sharing timing leakage: leaks prompts through timing variations in multi-tenant environments; this is a side channel rather than internal state analysis.
Linear probes and refusal direction
- Linear probes: an established technique for reading layer representations.
- Refusal direction research: shows that refusal in aligned chat models is mediated by a single direction in the residual stream, and per-layer probes can decode refusal before the final layer; this paper reproduces this property as a use case, not as a new interpretability finding.
Baselines and comparison
- The paper uses a dataset-tag rule and a prompt-text classifier as baselines, comparing refusal determination performance on the HarmBench and XSTest datasets.
Positioning difference: The authors claim this paper is the first forensic method targeting hidden states of the live inference process, distinguishing it from work on static artifacts or side channels; the probe results serve only as a use case for recovered states, not as a new interpretability claim.
How does the paper address this problem?
Using a six-stage pipeline (attach, clock, identify, read, validate, use), identify and read the residual stream from memory allocation events with a non-modifying debugger.
Overall approach: Propose a platform-independent six-stage pipeline, in which only the clock mechanism and identification rules are adapted per platform; use the PyTorch allocator or kernel-launch as the clock source, identify residual stream allocations from memory logs via a blind selector, read bytes after allocation, and analyze with linear probes.
Problem setting and formalization
- Goal: Obtain the per-layer residual stream from a running LLM process of unknown architecture, without modifying the target process and without relying on prior architectural knowledge.
- Formalization: Allocation a is linked to b if and only if size(a) = size(b) and nalloc(b) = nalloc(a) + ℓ(a) − 1, where ℓ(a) is the lifetime of allocation a (the number of allocations between birth and free).
Six-stage pipeline
- Attach: Use a CUDA debugger (cuda-gdb) to attach to an unmodified PyTorch process.
- Clock: Produce an event stream indicating the execution moment of each layer; the primary clock uses the PyTorch caching allocator to set non-stopping breakpoints at malloc/free, recording size and device pointer; the alternative clock breaks at each kernel launch.
- Identify: The blind selector has two steps: the primary rule identifies hand-off chains (a layer output is freed immediately after the next layer's output allocation), requiring at least 8 members; if none, fall back to regular constant-stride chains (at least 8 members, 90% of consecutive intervals equal to the stride).
- Read: Read bytes at one allocation after the allocation (since at this point the allocation precedes the kernel that fills it); for the last token, read the last row of hidden-size × element-size bytes.
- Validate: Validate offline against calibration models and hook-based references; validation logic is not run during collection.
- Use: Pass the recovered state to downstream analyses (e.g., depth prediction and refusal prediction probes).
Key design choices
- Non-modifying observation: Choose ptrace and the CUDA debugger over Python patching or LD_PRELOAD to maintain evidentiary neutrality, despite causing 46–192× slowdown.
- Blind selection: Do not input layer count or architecture; rely only on the temporal order and size patterns of memory allocations to identify the residual stream.
- Read timing: Determine by measurement that reading must occur after allocation; reading at the allocation point yields stale data.
Probe training and application
- Depth probe: Predict which layer the state comes from, trained with 16-token sentences.
- Refusal probe: Read the residual state at the last prompt token of chat-templated prompts, with the label being whether the model begins with a refusal phrase (standard list); train with logistic/softmax loss and weight decay, applied to standardized features.
What experiments did the paper conduct?
Byte-level reconstruction was validated on 33 models, the refusal probe AUROC 0.88 was validated on Qwen2.5-0.5B, and a 46–192× slowdown was measured.
Experimental setup
- Hardware and software: A single NVIDIA RTX 2080 Max-Q (8GB, sm_75), driver 580.178.04, CUDA 13.0, PyTorch 2.13.0, cuda-gdb 13.0, vLLM 0.32.0, SGLang 0.5.23.
- Models tested: 33 open-source decoder-only models (covering 9 architecture classes), plus 5 frozen rule models; specifically including Qwen2.5-0.5B-Instruct, TinyLlama, Llama-3.2-1B, GPT-2, Pythia, etc.
- Datasets: the P39 dataset with 228 recovery rows for development; Experiment A used 142 prompts from HarmBench and XSTest (never used for rule, probe, or threshold derivation); the attribution study used 228 prompts derived from JailbreakBench and AdvBench (41–43 tokens).
- Baselines: Dataset-tag rule (balanced accuracy 0.731) and prompt-text classifier (AUROC 0.559).
- Metrics: byte-exact match, AUROC, error rate, false positive rate (FPR), false negative rate (FNR), balanced accuracy, ECE.
- Evaluation method: pre-registered tests (script hash fixed in advance), hook-based reference as ground truth, offline bootstrap for 95% confidence intervals.
Main results
Swipe horizontally to read the table
Platform/configuration Clock Identification rule Result (per Table I) HF, static KV cache Allocator Two-step selector Byte-exact, 33 models; decode 6/7 vLLM, eager Allocator or kernel Selector; or stream class Byte-exact; stream 24/24, 22/22 vLLM, default (compile+graphs) Kernel launch Kernel-name dataflow rule Prefill 31/31, 23/23, 27/27 on 3 fresh models; decode 6/6 runs llama.cpp, CUDA backend Kernel launch Additive identity (prefill); norm-launch rule (decode) Prefill byte-exact; decode 10/10 runs; quantised Q8_0/Q4_K_M 12/12 runs SGLang Kernel launch Kernel-name rule (FusedAddRMSNorm) Prefill 6/6 runs; decode 4/6 (fails on Llama-3.2-1B) - The authors state that the two-step selector successfully identified the correct memory chain on 33 models, byte-exactly matching the reference state, consistent across three passes.
- In vLLM eager mode, all 6 models matched byte-exactly at every layer; under the default configuration the allocator clock failed, while the kernel-launch clock remained visible.
Refusal probe performance (Experiment A)
- Setup: Qwen2.5-0.5B-Instruct, vLLM default configuration, 142 untouched prompts (HarmBench + XSTest), frozen probe (layer 21, threshold logit 0).
- Results: AUROC 0.876 [0.808, 0.929], balanced accuracy 0.777, accuracy 0.803 (majority baseline 0.725), FPR 0.282, FNR 0.165, ECE 0.160 (Table III).
- By source: XSTest unsafe accuracy 0.945, HarmBench 0.778 (18 prompts), XSTest safe 0.696.
- Comparison: outperforms the dataset-tag rule (balanced accuracy 0.731) and the prompt-text classifier (AUROC 0.559).
Attribution error rates
- Setup: 228 vLLM-eager recovery rows, positive class is refused, threshold fixed at the training source.
- Results: JailbreakBench (71 prompts) error rate 15.5% [7.0, 25.4], FPR 13.3%, FNR 19.2%, outperforming the dataset tag's 26.8%; the other source (157 prompts) error rate 4.5% [1.3, 7.6], FPR 6.2%, FNR 3.3%, outperforming the tag's 8.3% (Table II).
Overhead and length robustness
- Latency: Qwen2.5-0.5B under vLLM batch 1, median prefill latency 17.5–17.8 ms in eager mode, 9.6–10.7 ms in default mode; with the collector attached, 157–163 seconds/prompt (about 15,000× unattached default).
- Length robustness: at token lengths of 34, 48, 100, 160, and 230, the rules held unchanged, with layer end matching within 2% for 30/30 prompts.
Other ablations and analyses
- Negative results: differential diffing of two VRAM dumps failed (a single merged region contained all 13 outputs); the post-pass image had only 1/13 outputs complete; reading at the allocation point failed for all layers.
- Derived properties: token count, hidden size, and layer count can be derived from the running process (Table IV); under vLLM default, 6/6 prefill and decode were exact; SGLang decode had an incorrect token count on Llama-3.2-1B (4/6).
- Alternative method failure: the name-agnostic content rule failed at the dev gate (2/9 logs, eager only).
What are some points that can be further explored?
The authors note that the method is limited by platform dependence, model scale, batch size and intrusiveness, and suggest future work to expand platform support and reduce overhead.
Limitations and future directions noted by the authors
- Platform dependence: The method depends on the PyTorch caching allocator and build-specific offsets and is not platform-independent; TensorRT-LLM, TGI and other GPUs were not tested (Section VII-A).
- Model scale and configuration: Currently supports up to 3.4B parameters (a single 7B 4-bit model tested); a 4-bit 7B MoE model was not tested due to an 8GB memory limit; limited to batch size 1, fixed prompt length, fp32/bf16 (fp16 on vLLM) (Section VII-A).
- Dynamic batching: Dynamic or continuous batching and variable-length inputs break the fixed pass cycle and were not tested (Section VII-A).
- Intrusiveness: The debugger halts the process at each allocation, causing 46–192× slowdown; attaching a GPU debugger is not forensically neutral and may contaminate evidence (Section VII-B).
- Strength of evidence: Behavioral-level inference (Level 3) is preliminary, relies on refusal states with a known error rate, and was tested on only a single model and two small held-out sources (Section VII-C).
- Selector robustness: The fixed selector required two revisions, with a risk of overfitting to test models; there is no platform-independent identification rule (Section VII-E).
Experimental coverage
- The tested models are 33 open-source decoder-only models (9 architecture classes) plus 5 frozen rule models, with parameter scale up to 3.4B (except one 7B 4-bit).
- Platforms covered are Hugging Face (static KV cache), vLLM (eager and default), llama.cpp (CUDA backend, including Q8_0/Q4_K_M quantization), SGLang (Triton attention); TensorRT-LLM and TGI were not tested.
- The refusal probe evaluation was conducted on Qwen2.5-0.5B-Instruct, using 142 prompts from HarmBench and XSTest (pre-registered), and 228 prompts from JailbreakBench/AdvBench/Alpaca (exploratory).
- The paper reports no tests across GPU architectures (other than sm_75), no experiments with batch size >1, and no performance under dynamic batching.
Summarize the main content of the paper
Propose a method to obtain the residual stream from a live GPU process using a non-modifying debugger, validate byte-level recovery on 33 models, and use linear probes to achieve request-level refusal attribution.
This paper addresses the problem in LLM service forensics that the transient hidden states of a single request cannot be obtained, and proposes a six-stage non-modifying debugging pipeline. The method uses the PyTorch allocator or kernel launches as a clock, identifies the residual stream from memory allocation logs via a blind selector, and reads bytes from VRAM without modifying the target process.
Experiments achieve byte-level reconstruction on 33 open-source decoder-only models (covering 9 architecture classes), spanning the Hugging Face, vLLM, llama.cpp, and SGLang platforms. On Qwen2.5-0.5B-Instruct, a refusal probe reaches AUROC 0.88 on 142 held-out prompts, with error rates of 5–15%, outperforming the dataset-tag baseline.
The authors claim the method provides a new evidence source for request-level attribution, but emphasize its intrusiveness (46–192× slowdown) and platform dependence, and that it is currently limited to small models (≤3.4B) and batch size 1 scenarios.