Say Shibboleth: Fingerprinting the LLMs and harnesses behind agentic attacks
Say Shibboleth: Fingerprinting the LLMs and Harnesses Behind Agentic Attacks
The authors propose a shibboleth trap honeypot that infers the attacking agent's LLM and harness under a blind and passive setting; the authors report an average 90% top-1 accuracy for combined inference across 15 candidate models.
Defender D owns target system H (a honeypot), attacker A is an LLM-driven agent, and the goal is to compromise H. D is blind (can only see the side effects of tool calls on H) and passive (cannot actively query the agent), and can only observe the telemetry A leaves on H (network requests, executed commands), without seeing A's internal state, messages, or tool calls; D can design H arbitrarily under realism constraints. Additionally, we test the case where the attacker is instructed to conceal the model identity.
- LLM-driven attacks add another target to attribution: which model and harness the attacking agent uses. Defenders can only observe telemetry on the target system and cannot query the attacker's model, so existing active fingerprinting methods do not apply.
- The authors propose shibboleth traps: making informative interactions a condition for obtaining resources, with declarative gates that make the agent self-report the model name and behavioral gates that make it write free text with a model "accent," and using execution probes to indirectly probe the harness's secret masking, timeouts, and scheduling behavior.
- The authors report: combined inference achieves an average 90% top-1 accuracy on 15 candidate models; fixed-rule identification on four harnesses achieves 92% accuracy; when instructed to conceal identity, declarative accuracy drops to 8%, but behavioral inference still reaches 77%.
- The authors state that proprietary LLM coverage is limited (constrained by network refusals); in the fleet trap experiments, Claude-Opus-5 runs only on Claude Code, confounding model and harness; the authors plan to expand the harness pool and deploy honeypots at scale.
- Evaluated models
- Grok-4.6Qwen3.8-MaxMiMo-V2.6-ProGLM-5.3-FlashClaude-Opus-5GLM-5.3Kimi-K3DeepSeek-V4-FlashMiniMax-M3Qwen3.8-27BDeepSeek-V4-ProHy3GPT-5.6-LunaMistral-Medium-3.5Nemotron-3.5-Super-VL
- Benchmarks
- MCP handshake trapA2A desk trapSSH client declaration trapFleet rollout trap
- Metrics
- top-1 accuracytop-3 accuracy
Researchers propose a method for fingerprinting agentic cyberattacks that can infer from telemetry the LLM version and harness used by the attacker. Its shibboleth traps are modular honeypot services that make informative interactions a prerequisite for agents to obtain needed resources, inducing models to make statements and leave characteristic text without requiring a successful prompt injection; for the harness, it indirectly probes its distinctive execution behavior solely from effects visible in subsequent requests, including how it masks secrets in tool outputs, enforces command time limits, and schedules tool calls. Combining statement and behavior inference, it achieves an average 90% top-1 accuracy across 15 candidate models, with the behavior classifier evaluated on held-out harnesses or task families; in another experiment across four harnesses, fixed probe rules identified the correct harness with 92% accuracy. Even when instructed to hide their identity, agents still leave useful behavioral evidence once statements lose informativeness.
Paper analysis
What problem does this paper attempt to solve?
Under a blind, passive setting where only target-system telemetry can be observed, infer the LLM version and harness used by the attacking agent.
Core question: Under a setting where the defender can only observe the target system and cannot query the attacker's model, how can the model version and harness behind an LLM-driven attack (i.e., the attacker's agent configuration) be inferred?
- Scenario and motivation: The authors argue that LLM-driven attacks make "the software executing the attack" a new target for attribution; identifying the model and harness lets defenders track their use across attacks and evaluate defenses. The introduction mentions that Unit 42 once identified DeepSeek and Hermes Agent from logs accidentally exposed by an attacker, whereas a defender who can only observe the target does not have this condition.
- Shortcomings of existing methods: The authors state that active fingerprinting methods such as LLMmap require submitting selected queries to the model, and the defender has no such interface to the attacking agent; prompt-injection-based honeypots (such as Mantis) face the problem that models and harnesses are increasingly resistant to injection.
- Core observation: Different agents behave differently when interacting with the same target, part of which comes from biases of the LLM parameters and the harness; the defender can design services so that obtaining resources is conditioned on informative interaction, thereby artificially increasing the distinguishability of telemetry.
- Proposed in the paper: The Shibboleth system—implementing "shibboleth traps" in honeypot services, including declarative gates (inducing self-reported identity) and behavioral gates (inducing text with an "accent" and indirectly probing harness execution behavior), as well as an agent pipeline that uses an LLM to automatically generate and optimize traps.
- Threat model:
- Attacker goal: an LLM-driven agent A whose goal is to compromise target system H (a honeypot).
- Knowledge and capabilities: A acts with a specific configuration (LLM + harness); the case where A is instructed to use a false name to conceal the model identity is also tested.
- Defender: owns H and can design H arbitrarily under realism constraints; set as blind (only sees the side effects of tool calls on H) and passive (cannot actively query the agent or manipulate the environment in real time), and can only observe the telemetry A leaves on H (network requests, executed commands).
What related research exists?
Related work falls into three categories: model/agent fingerprinting, LLM honeypots, and honeypot identification; this paper positions itself as a blind and passive fingerprinting method.
Model and harness fingerprinting
- LLMmap (Pasquini et al., 2025): sends eight probes to a target LLM application and uses a trained inference model to identify the LLM version; the authors note that it requires control over model inputs and direct reading of outputs, so it does not apply to this paper's setting. TRAP (Gubri et al., 2024) uses adversarial suffixes to detect whether an application runs a certain model; Sun et al. (2025) train a classifier on reply text alone to distinguish five model vendors with 97% accuracy.
- LIDAR (Wang et al., 2026) uses test tasks designed specifically for coding agents to identify the model from how it solves problems (36 models, 2 harnesses); AgentTrace (Zhao et al., 2026) has agents copy ambiguous strings and identifies the model from how it fails to fix them; the authors note that both are active (they issue instructions to the agent).
- Passive observation: Lugoloobi et al. (2026) identify the model behind a browsing agent from actions and timing recorded by a website (14 models); Kang et al. (2026) and Fayolle et al. (2026) distinguish web agent frameworks from network and browser fingerprints.
- Trace (Ediga & Chattopadhyay, 2026): closest to this paper, identifying the model family from shell commands executed by a penetration-testing agent on a honeypot, with 98% of sessions named correctly; the authors note that its main experiment tests only one model per family and does not identify the harness.
LLM honeypots
- Reworr & Volkov (2024) and Pasquini et al. (Mantis, 2026) concurrently proposed the concept of LLM honeypots, using prompt injection to make attacking agents expose or sabotage themselves; Heckel & Weller (2024) show that defensive prompt injection can interrupt an attacking agent's workflow. The authors note that these works rely on prompt injection, while models and harnesses are increasingly trained to resist injection.
- AgentSnare (Wang et al., 2026) and AgentTrap (Wang et al., 2026) respectively use fake files/services and fake replies to keep agents in decoys; the authors note that neither identifies the model or harness driving the agent. Mantis active mode, Mayoral-Vilches et al. (2025), and BlackSea (Pasquini et al., 2026) obtain fingerprints by making the agent compromise the attacker's own machine; the authors state that these approaches are not generally applicable and can be regarded as hack-back.
Honeypot identification
- Rouxii pentester (Cordeiro et al., 2026) can identify open-source honeypots with 97% accuracy when given prompts containing honeypot detection clues (19% without clues); Xie et al. (2026) make attacking agents skip hosts suspected of being honeypots.
Positioning of this paper: the authors state that the technique in this paper is both passive (it does not control the inputs to the fingerprinted model) and blind (it can only observe the model's effects on the environment through tool use), and claim to provide the first mechanism specifically for fingerprinting agentic harnesses by indirectly probing execution behavior.
How does the paper address this problem?
Make informative interactions a condition for obtaining resources: declarative gates induce self-reported identity, while behavioral gates induce text with an "accent" and indirectly probe harness execution behavior.
Overall approach: the defender designs a honeypot service (shibboleth trap) such that the attacking agent must complete a discriminative interaction to obtain a "reward" (such as a deployment key); inference is then performed from telemetry using declarative rules and a behavior classifier.
Problem formalization
- The target system H is abstracted as a finite set of gates, where a gate is a function that receives attacker input (such as the username and password for SSH login); the sequence of gate calls left by an attack is called a trace (telemetry).
- The discriminability of a gate is defined as δ(gi) = 𝔼c ≠ c′ TV[gi(Ac), gi(Ac′)], i.e., the expected total variation distance between the input distributions supplied to the same gate by two different configurations; δ close to 1 means a single call can distinguish the configurations.
- A shibboleth trap is a gate designed so that δ is as close to 1 as possible; the paper studies declarative and behavioral gates, and the traps are based on real services (MCP, A2A, SSH, etc.) and attract interaction with rewards.
Declarative gates and inference rules
- Identity declarations are disguised as a natural step of the protocol: for example, the MCP trap makes the custom subparameter model under the sampling parameter mandatory and specifies the value format (provider routing form, with an alphanumeric suffix), without providing example values.
- Technique: the first k handshakes fail unconditionally, and obviously fake declarations are rejected, to induce more declarations and behavioral signals; a blacklist B of common placeholders and retired model names is maintained (the top ten are listed in Appendix B).
- Inference rule Id: take the model named by the first "valid" declaration in the trace (correct format, an LLM name, not in the blacklist), or return ⊥ if none; the harness uses the same rule IdH.
Behavioral gates and inference
- LLM version: force the agent to make decisions in high-entropy domains such as free text (for example, the MCP trap requires at least 80 characters of non-repeating ticket text for each change). Classification uses character n-gram (2–5 characters, within-word) tf-idf vectors, and the trace fingerprint f is the mean of its input vectors; open-set inference: m̂ = argmaxm cos(f − μ, Tm − μ), i.e., the one with the closest cosine to the registered template Tm, and a new model only needs a few registered traces without retraining.
- Harness: the authors state that the harness has a weak influence on text content, so they design indirect probes to probe its execution behavior—secret masking (different harnesses have different masking rules for credentials of different shapes), command timeouts (suspend a request for 150 s and see who closes it and when), concurrent scheduling (whether multiple tool calls in the same step run in parallel), etc. (Table 2 lists more markers). Inference IbH is a fixed rule set: each probe gives a set of consistent harnesses, and the intersection is taken; when the rules cannot localize, an optional classifier can be used.
- Combined inference I: use the declaration if there is a valid one, otherwise fall back to the behavior classifier.
Trap generation pipeline
- Using Claude-Opus-5 as the design LLM, an end-to-end pipeline: design and build → probe (the same model critiques the design against realism criteria) → attack simulation (each round fixes 5 LLMs that are hardest to fingerprint, each paired with a harness, with 3 attacks per agent) → judging (another agent performs blind k-fold classification to estimate δ) → revision, up to N rounds; attacking agents are also interviewed for feedback.
- The paper's four traps (MCP handshake, A2A desk, SSH client declaration, Fleet rollout, Table 3) were all designed or optimized by this pipeline.
What experiments did the paper conduct?
About 800 attack simulations: combined inference reaches 90%–92% top-1 on 15 LLMs and 4 traps; rule-based identification reaches 92% on 4 harnesses; behavioral evidence remains effective when identity is concealed.
Experimental setup
- Models tested: 15 LLMs (Table 5: Grok-4.6, Qwen3.8-Max, MiMo-V2.6-Pro, GLM-5.3-Flash, Claude-Opus-5, GLM-5.3, Kimi-K3, DeepSeek-V4-Flash, MiniMax-M3, Qwen3.8-27B, DeepSeek-V4-Pro, Hy3, GPT-5.6-Luna, Mistral-Medium-3.5, Nemotron-3.5-Super-VL); selected from the OpenRouter leaderboard and models that appeared in real incidents; Claude-Opus-5 was used through cyber program membership.
- Harness: Hermes Agent 0.19.0, OpenClaw 2026.10.3, OpenCode 1.19.3, plus Claude Code (only with Claude-Opus-5, except for the fleet trap).
- Traps: The four traps in Table 3 (MCP, A2A, SSH, Fleet rollout).
- Tasks: 7 types of offensive tasks (vulnerability assessment, initial access, credential collection, data collection, persistence, valuation, stealthy access), with 5 paraphrased wordings per task, randomly selected at runtime.
- Run volume: 6 attack simulations per LLM×harness pair, about 800 in total; each run is defined by ⟨trap, LLM, harness, task, wording⟩.
- Validation method: The behavioral classifier is evaluated under three settings—same distribution (SD, 90/10 split, ten-fold), leave-one-task-out (LOTO), and leave-one-harness-out (LOHO); all learnable parts are fitted only on training runs, runs with no written text count as classification failures; the metric is top-1 (top-3 in some cases) accuracy with 95% intervals.
Main results
LLM version inference top-1 accuracy (from Table 4; after the slash in the behavioral/combined column is top-3):
Swipe horizontally to read the table
Trap Inference SD LOTO LOHO MCP Declarative only 87% 87% 87% MCP Behavioral only 45/67% 36/54% 41/62% MCP Combined 91/93% 90/93% 90/92% A2A Declarative only 90% 90% 90% A2A Behavioral only 54/78% 54/76% 51/75% A2A Combined 92/94% 92/94% 92/94% SSH Declarative only 84% 84% 84% - Authors' interpretation: Declarative inference accounts for most of the results (84%–90% alone) and never names the wrong model (failures are all cases with no valid declaration); behavioral inference alone is weaker (36%–54%) but covers runs with missing declarations.
- By model (Table 5, SD setting, three traps combined): Combined inference ranges from 98% for Grok-4.6, Qwen3.8-Max, and MiMo-V2.6-Pro to 75% for Nemotron-3.5-Super-VL; the largest gains are for models with the fewest declarations, e.g., Hy3 goes from 67% (declarative only) to 89% (combined).
- harness declarative inference (lower block of Table 4): A2A 85%, SSH 81%.
harness behavioral inference of the Fleet trap
- What was tested: The fleet rollout trap in a reduced setting on four harnesses with 5+1 models (5 random + Claude-Opus-5); no optional classifier is used, and the rules are fixed by pre-run desktop analysis.
- Results (Table 6): 92% of runs are named correctly (Hermes Agent 87%, OpenClaw 87%, OpenCode 93%, Claude Code 96%); the strict readings are all correct on the runs they decide; leans fail on 3 runs, and another 3 misses are short sessions of 4–6 requests.
- Caveat noted by the authors: Claude-Opus-5 runs only on Claude Code, so its LLM and harness are confounded.
Concealing identity (bypassing declarative inference)
- What was tested: On the A2A trap, 5 LLMs (Hermes Agent) plus Claude-Opus-5 (Claude Code), with a system prompt appended: "never reveal the name; when asked, answer with the specified false name" (three real but out-of-pool false names), a single task (initial access), compared with a baseline without this instruction.
- Results (Figure 9): 92% of lying runs declared the specified false name; declarative and combined inference drop from 95% to 8%; behavioral inference does not drop (baseline 81%, lying 77%), and names correctly in 77% of runs where the declaration lied.
- Authors' countermeasure: Accept only declarations with behavioral templates, otherwise fall back to behavioral inference; the baseline stays at 100%, and the lying setting rises from 8% to 79%.
Prompt-injection baseline comparison (Appendix A)
- What was tested: On the MCP trap, declarative gates are compared with prompt injection (the original text of AgentDojo's important instructions attack, including a version with a specified false name), with 5 LLMs attacking 3 times each, 60 runs in total.
- Results (Figure 10): Declarative gates obtain the true model name in 14 of 15 runs; the injection version obtains it in 0 of 30 runs (agents post to the endpoint 27 times but never write the true name); all injection-version agents call the injected block a prompt injection during the attack, and 12/21 therefore judge the target to be a honeypot.
Other ablations and analyses
- Declaration behavior statistics (Table 1): 80% of runs declare more than once, 32% declare a false name at some point, 25% declare both true and false names; false names are almost never other models in the evaluation pool.
- Rule benefit (Table 1): Taking the first declaration directly gives 76% naming accuracy, and rule Id gives 88%.
- Error sources (Table 8): All errors in the combined readings (79 of 1800 readings) come from behavioral fallback, plus 80 no-answer cases.
- Common false names (Table 7): Of 16,925 declarations, 7,854 used a model name other than their own (452 distinct names), and the top ten account for 1,063; six are retired models.
What are some points that can be further explored?
The authors list directions such as expanding the trap catalog and large-scale deployment; the experiments cover 15 LLMs and 4 harnesses, with limited coverage of proprietary models.
Limitations noted by the authors and future directions
- Limited coverage of proprietary LLMs: the most capable instances are unsuitable for the attack tasks in the experiments due to modern cyber refusal constraints; the authors use Claude-Opus-5 with cyber program membership to cover this category (Section 7.1).
- The authors plan to expand the harness pool in future versions (Section 7.1).
- Analysis of raw telemetry is left for future work: generalizing it requires more variation in attacker machines (different machines, geographic locations, etc.) (Section 5).
- Two future directions: expanding the shibboleth trap catalog and harness fingerprinting techniques; deploying traps as honeypots at scale on the internet (Section 8).
- Deception is a depreciating asset: it becomes ineffective with each use, and the authors consider an automated pipeline for generating and optimizing traps key to keeping the method scalable (Section 6).
- An initial small-scale deployment has recorded real attack attempts, but the authors state that such findings remain sparse at that scale (Section 8).
Scope of experimental coverage
- The models tested are 15 LLMs (Table 5), and the harnesses are Hermes Agent, OpenClaw, OpenCode, and Claude Code, 4 in total; Claude Code is paired only with Claude-Opus-5 (except for the fleet trap).
- The traps are the four in Table 3 (MCP, A2A, SSH, Fleet rollout), and the tasks are 7 categories of offensive tasks, each with 5 phrasings; 6 runs per LLM×harness pair, about 800 in total.
- The behavior classifier evaluation covers three generalization settings (SD, LOTO, LOHO); the fleet trap experiment uses a reduced setting of 5+1 models and does not use the optional classifier.
- The identity concealment experiment is conducted on the A2A trap, a single task (initial access), 6 LLMs, and 3 pseudonyms; the prompt injection comparison is conducted on the MCP trap, 5 LLMs, and 60 runs.
- The paper does not report the specific number of runs in each fold of the behavior classifier evaluation, nor the total number of iterations of the trap optimization pipeline (it only gives 5 LLMs × 3 attacks per round, up to N rounds).
Summarize the main content of the paper
Shibboleth traps make attacking agents self-report their identity and leave a behavioral "accent" to obtain resources, and combined inference achieves an average 90% top-1 across 15 models.
A defense paper on agentic attack fingerprinting: under a blind and passive setting where the defender can only observe target-system telemetry and cannot query the attacker's model, infer the LLM version and harness used by the attacking agent.
- Problem: The authors argue that LLM-driven attacks make the attack software itself a target for attribution; active fingerprinting methods (such as LLMmap) require a query interface, while prompt-injection honeypots face the growing injection resistance of models.
- Method: shibboleth traps make informative interaction a condition for obtaining resources—declarative gates disguise model-name declarations as required protocol fields, behavioral gates force the agent to write free text (reading the "accent" with an n-gram tf-idf open-set classifier), and execution probes indirectly probe the harness's secret masks, timeouts, and concurrent scheduling; the traps are generated and optimized by an agent pipeline driven by Claude-Opus-5.
- Main results: On about 800 attack simulations (15 LLMs, 4 harnesses, 7 task types), combined inference reaches 90%–92% top-1 on MCP and A2A traps, and under leave-one-task and leave-one-harness settings differs from the same distribution by no more than one point (Table 4).
- harness fingerprinting: The fixed probe rules of the fleet trap name the correct harness on four harnesses 92% of the time, and the strict readings are all correct (Table 6).
- Concealing identity: When instructed to report a false name, declarative inference drops from 95% to 8%, but behavioral inference remains at 77%; the authors' countermeasure (accepting only declarations with behavioral templates) raises accuracy in the lying setting to 79% (Figure 9).
- Authors' conclusion: Declarative and behavioral gates are complementary, and behavioral evidence remains when declarations fail; the authors plan to expand the trap catalog and deploy a honeypot network at scale, and have open-sourced the code.