Can LLMs fool graph learning? Exploring universal adversarial attacks on text-attributed graphs

How to design a universal adversarial attack on TAGs that transfers across GNN and LLM architectures under black-box settings, to evaluate the security of text-attributed graph learning.

• Scenario: TAGs use both text and topology on nodes. The authors believe this fusion increases expressiveness but also exposes a new adversarial surface through the text modality; backbones include both GNN reasoners and LLM reasoners, requiring universal attacks to check robustness and reduce misuse risk.
• Existing shortcomings: The authors state existing attacks usually modify edges or node features, making them hard to explain and manipulate text; embedding-space attacks are impractical and hard to explain; text-level injection is still weaker than embedding-space attacks. The introduction also states graph attacks are weaker in LLM-driven graph learning.
• Hypothesis: The authors believe LLMs can generate context-relevant but misleading content and thus can serve as graph attackers; the key is making topology edits and text edits point to the same wrong label without gradients.
• Method: Propose BadGraph: retrieve target influencers, construct cross-modal aligned attack shortcuts, then use an LLM to jointly perturb adjacency and node text. The authors also give a homophily preservation upper bound and the Cross-Modal Shortcut Theory.
• Threat model:
  • Goal: Construct G'=(A', S') so the model outputs the predetermined wrong label Y'.
  • Knowledge: black-box evasion, no access to architecture, parameters, training process, or text encoder.
  • Capability: only architecture-agnostic universal attacks at test time; edge edits do not exceed b_A, text edits do not exceed b_S.
  • Victim system: GNN reasoners and LLM reasoners in TAG node classification, and the same perturbed graph must transfer across heterogeneous backbones.

According to the introduction and Related Work, related work can be divided into three groups.

Graph adversarial attacks

• Evasion and poisoning: Nettack (Zügner et al., 2018) uses a greedy strategy to modify edges; Li et al. (2024) extend black-box evasion to interpretability settings. On the poisoning side, SBAC (Dai et al., 2024) injects semantic trigger nodes, EPD (Liu et al., 2025) distinguishes benign augmentation from adversarial manipulation, and GOttack (Alom et al., 2025) uses orbit structures to manipulate and attack GNNs. The authors claim these works ignore text-based attack interfaces in the LLM era.
• Structural or feature perturbation: The authors claim that attacks modifying edges or node features under budget constraints struggle to handle TAG text, consistent with findings by Guo et al. (2024) and Zhang et al. (2024) that graph attacks are weaker in LLM-driven graph learning.

LLM-generated attacks

• Prompts and other domains: PromptAttack (Xu et al., 2023) uses adversarial prompts to induce target LLMs to produce harmful outputs; PAIR (Chao et al., 2023) uses an attacker LLM to iteratively generate jailbreak prompts. Attack-in-the-Chain (Liu et al., 2025) attacks neural ranking with chain of thought; CheatAgent (Ning et al., 2024) uses an LLM agent to attack recommender systems.
• Text injection on graphs: WTGIA (Lei et al., 2024) performs text-level injection. The authors claim its effectiveness decreases as text interpretability increases, and it is weaker than embedding-space attacks.

Two types of reasoners on TAGs

• GNN reasoner: First encodes node text with a language model, then propagates through a GNN. The authors mention encoding shifting from TF-IDF to OFA (Liu et al., 2023) and TAPE (He et al., 2024).
• LLM reasoner: Rewrites the graph into a form processable by an LLM, prompting a frozen LLM (Chen et al., 2024), or retraining graph-related tokenizers (Chen et al., 2024, LLaGA). The authors believe the key is how to describe the TAG to elicit graph reasoning.

Baselines and data

• Baseline methods: RND, FLIP, STACK, PGD, NETTACK, SGAttack, WTGIA. In comparisons, the attacker is assumed not to know the embedding technique; baselines uniformly use BoW features, a GCN surrogate, and at most two edge perturbations per node (Appendix A.3, A.4).
• Benchmarks/datasets: Cora, OGBN-Arxiv, OGBN-Products.

The authors claim that, to their knowledge, this paper is the first general TAG adversarial attack across multiple graph learning models; the difference is using an LLM to simultaneously perturb topology and text under black-box settings, rather than only modifying structure, features, or performing text injection.

BadGraph采用检索后再推理：先找语义远离的influencer，再让LLM生成与之对齐的拓扑和文本扰动，使同一扰动图可接到不同TAG backbone。

攻击目标

式(1)要求 f_θm^(k)(A′, S′) → Y′, m ∈ {GNN, LLM}。含义是同一扰动图要让第k个GNN或LLM reasoner输出预定错误标签，边与文本的编辑次数分别不超过b_A、b_S。论文把每节点拓扑预算设为b_Ai=2，即一次加边和一次删边。

目标influencer检索

• 动机：作者称无梯度时若对所有节点对询问LLM，查询量为O(V^2)，且缺少图先验，难以定位有效扰动区域。
• 检索：用GraphEncoder把图编成节点嵌入，编码器可以是GCN、GAT等GNN。对目标节点按余弦不相似度取top-K个最不相似节点，作为influencer集合。附录A.4写明K=5；Cora与Products用GAT，Arxiv用GIN。
• 作用：这些节点提供指向错误标签的捷径先验，把后续LLM推理限制在小集合上。作者称表达力更强的编码器会带来更强、更可迁移的扰动。

拓扑扰动

• 引出图知识：对目标节点的1-hop邻域和节点文本做结构化提示，得到邻接原因摘要，作为后续编辑的依据。
• 删边：在1-hop邻居中选择与目标语义最相关的节点并删除该边，以削弱支持原标签的局部结构证据。
• 加边：在influencer集合中选择与目标最不相关的节点并连边，把局部连接引向该influencer的标签。作者称这把边决策复杂度从O(N^2)降到O(N)。

文本扰动

• 同一锚点：文本侧沿用加边所选的同一个influencer，先从其特征文本中抽出一个能反映其类别的关键词。
• 改写约束：在文本预算b_Si内修改目标节点文本，保留部分原有词语，并纳入该关键词，使语义偏向influencer的类别。Cora与OGBN-Arxiv的节点文本是论文标题；OGBN-Products用商品名，缺名时用商品描述。
• 查询：作者称每个目标节点只需两次LLM查询。主实验攻击者模型为DeepSeek-V3，经API生成拓扑与文本扰动。正文给出了分步提示的作用，此处不复述其句式。

理论解释

• 同质性上界：假设文本编码器Lipschitz连续。作者给出全局同质性变化被边扰动比与文本嵌入漂移共同上界约束（式11），用来解释小预算下扰动不易从全局同质性看出。
• 跨模态捷径：结构捷径把目标连到influencer，文本捷径把节点文本推向同一节点。作者将联合效应写成 Δ_joint ≈ Δ_δA,δS > Δ_δA+Δ_δS，即对齐的跨模态扰动强于两个单模态之和。

Experimental Setup

• Data: Cora has 2708 nodes, 5429 edges, 7 classes; OGBN-Arxiv has 169343 nodes, 1166243 edges, 40 classes; OGBN-Products subset has 54025 nodes, 74420 edges, 47 classes (Table 6). Cora target nodes use the medium split, the paper does not give the count; Arxiv samples 1000 from the medium split, Products samples 1000 from the hard split. The first two use paper titles, Products uses product names, and descriptions are used when names are missing.
• Models under test: GNN reasoners are R-GCN, GIN, GraphSAGE, TAGCN, GCN, with encodings TF-IDF, SBERT, TAPE. LLM reasoners are DeepSeek-V3 and Mistral-7B, zero-shot, with input text and 2-hop neighborhood. The appendix also reports SGCN, RUNG; Table 5 reports Qwen-Plus accuracy.
• Attack side and baselines: The main experiment's attacker LLM is DeepSeek-V3; retrieval K=5, Cora and Products use GAT, Arxiv uses GIN. Baselines are RND, FLIP, STACK, PGD, NETTACK, SGAttack, WTGIA. Baselines use BoW, GCN surrogate, at most two edge perturbations per node. WTGIA injects 20 nodes for Cora, 166 for Products, 166 for Arxiv, with text transcribed by LLaMA3-8B from embeddings.
• Metrics: The main metric is node classification Accuracy, where lower means stronger attack. There are also 3-Max Accuracy (averaged over the three most robust defenses) and Weighted Accuracy weighted by defense robustness (6.1.4). Figure 4 uses ASR, the paper does not give the formula.
• Replication and protocol: Each experiment is run five times; the main text says mean and standard deviation are reported, but the tables do not list standard deviations. All attacks are evaluated under the same black-box setting, and the attacker does not know the embedding technique. In Table 1, several cells for SGAttack and WTGIA are "-".

Main Results

Swipe horizontally to read the table

Model	Condition	Clean	BadGraph	Source
DeepSeek-V3	Cora, LLM reasoner	71.85	17.03	Table 2
DeepSeek-V3	Products, LLM reasoner	77.10	55.80	Table 2
Mistral-7B	Arxiv, LLM reasoner	9.30	2.90	Table 2
GIN	Arxiv, TAPE	77.68	25.22	Table 1
GraphSAGE	Cora, SBERT	86.00	39.93	Table 1
GIN	Cora, TF-IDF	85.93	63.04	Table 1
R-GCN	Arxiv, TAPE	70.00	65.22	Table 1
R-GCN	Products, TAPE	89.54	80.52	Table 1

Lower accuracy means stronger attack. GCN, TAGCN, SGCN, Qwen-Plus, and the remaining encodings and Mistral-7B Cora and Products rows are omitted; the table retains the Arxiv TAPE R-GCN and Products TAPE R-GCN rows with higher BadGraph accuracy from Table 1. Arxiv in Table 1 is OGBN-Arxiv in the main text.

• Authors' summary: The authors state that relative to baselines, BadGraph achieves SOTA on all tested target models and datasets. The abstract and 6.2.2 state that on Cora, DeepSeek prediction accuracy drops relatively by 76.3%, and WTGIA drops relatively by 11.3% under the same setting. 6.2.1 also states GNN scenarios drop by nearly 40% (Cora), 52% (OGBN-Arxiv), and about 30% (OGBN-Products), without specifying whether these are relative drops or percentage points.
• Feature expressiveness: The authors state that richer node features are more robust, traditional structural attacks are more effective against TF-IDF, have less impact on SBERT and TAPE, and on OGBN-Arxiv some attacks increase accuracy. In Table 1, STACK for Arxiv, TAPE, R-GCN is 75.82, higher than Clean 70.00. Under BadGraph in the same table, Arxiv, TAPE, GIN is 25.22, lower than 41.16 for that model under TF-IDF.
• Single modality: The authors state that modifying only text or only structure is weaker. In Table 2, for Cora, DeepSeek-V3, Our-text is 25.55, Our-struct is 67.03, Clean is 71.85. The authors state LLM reasoners are more susceptible to text attacks and give examples of text-only relative drop of 64% and structure-only relative drop of 6.3%; in the same table, for Arxiv, Mistral-7B, Our-struct is 4.70 and Our-text is 4.90.

Stealth and Overhead

• Homophily numbers: Table 3 on Arxiv, DeepSeek reasoner: at 0 nodes, Acc is 64.6%, edge homophily 0.8184, node homophily 0.4257; at 1000 nodes, perturbation ratio 0.17%, Acc is 22.9%, edge homophily 0.8182; at 1500 nodes, 0.26%, Acc is 22.6%, edge homophily 0.8181, node homophily 0.4316. The authors state global homophily is almost unchanged, and edits are paired local edge changes.
• Figures: The authors state that in Figure 2, WTGIA shifts the Cora edge homophily distribution more, while BadGraph only has local small changes. The authors state that in the T-SNE of Figure 3, BadGraph causes greater boundary distortion and class entanglement than WTGIA. The transcribed text does not give the values in the figures.
• Cost: The authors state retrieval training complexity is O(2KNd^2), and after retrieval, the LLM edge decision per target node is O(1); the whole process uses API calls on CPU, averaging about $0.0009 per node.

Prompts, Attacker Model, and Anchors

• Prompt rewriting: Table 4 compares prompt variants on Cora, the title does not state node encoding. Under Rephrasing, R-GCN, GCN, GIN, GraphSAGE, TAGCN are 58.15, 58.74, 50.74, 39.04, 54.59 respectively; under BadGraph they are 56.67, 56.00, 47.04, 39.93, 51.19. The authors state accuracy fluctuation is within 3.7%.
• Anchor inconsistency: Table 4's Anchor Mis. values are 71.63, 63.93, 71.04, 62.59, 77.11 respectively. The authors state that after cross-modal sharing of the same influencer is removed, attack effectiveness drops by up to 22.66%.
• Attacker LLM: Table 5's attackers are LLaMA-4-17B, Qwen-Plus, DeepSeek-V3, the title does not state encoding. Under Ours-DeepSeek, GIN is 47.04, GraphSAGE is 39.93, SGCN is 55.63, TAGCN is 51.19, Qwen is 16.29, DeepSeek is 16.29. The authors state different attacker LLMs are all effective, and larger models perturb more precisely.

Accuracy under Defenses

• RUNG: Table 7, Cora. Under TF-IDF, Clean is 86.81 and BadGraph is 62.44; under SBERT, Clean is 85.45, BadGraph is 35.11, WTGIA is 85.11, NETTACK is 83.56. The authors state accuracy drops by more than 50% under SBERT and state cross-modal alignment misleads RUNG's edge reweighting.
• Adversarial training: Table 8, Cora, encoding not stated in the paper. Under BadGraph, R-GCN, GCN, SGCN, TAGCN are 72.59, 75.19, 75.19, 75.56; relative to vanilla, ΔAcc is +1.70, +3.86, +3.93, +1.63. The authors state attacks remain effective after adversarial training.

Other Ablations and Analyses

• Encoder (Table 9, Cora, title does not state encoding): Under GCN encoder, GIN is 55.44 and DeepSeek is 21.11; under R-GCN encoder, GraphSAGE is 38.52 and DeepSeek is 19.59; under GAT encoder, GIN is 47.04 and DeepSeek is 17.03. For GraphSAGE, R-GCN encoder 38.52 is lower than GAT's 39.93.
• Node degree: Figure 4 compares Medium vs Hard and SBERT vs TF-IDF on OGBN-Products. The authors state low-degree nodes have higher ASR; the values in the figure are not given in the text. The main experiment's Products target nodes come from the hard split.
• Case study: 6.3.4 uses DeepSeek to show step-by-step reasoning for structural attacks and gives natural language explanations; specific node text is not restated.
• Aggregate metrics: Appendix Table 10 gives Average, 3-MAX, Weighted. The authors state BadGraph is still best on these aggregates; after transcription some columns may be misaligned, so misaligned cells are not cited here.
• Cells above Clean: In Table 1, FLIP for Arxiv, TAPE, GIN is 78.48, Clean is 77.68; STACK for Products, TAPE, R-GCN is 89.58, Clean is 89.54. In Table 2, Our-struct for Products, Mistral-7B is 12.80, Clean is 12.20.

Limitations and future directions noted by the authors

• The paper does not have a dedicated Limitations section, and the Conclusion does not frame any experimental result as a shortcoming of the current method.
• Future directions: The Conclusion states that this work lays the foundation for future research, including autonomous LLM agents for adaptive attacks (Conclusion).

Experimental coverage

• Data: Experiments use Cora, OGBN-Arxiv, and a subset of OGBN-Products; sizes are in Table 6; node text is paper titles or product names/descriptions (A.1).
• Systems under test: GNN reasoners include R-GCN, GIN, GraphSAGE, TAGCN, GCN, encoded with TF-IDF, SBERT, TAPE; the appendix also reports SGCN. LLM reasoners are DeepSeek-V3 and Mistral-7B, zero-shot, using text and 2-hop neighborhoods; Table 5 also reports Qwen-Plus accuracy.
• Attack configuration: Set to black-box, test-time. The main experiments use DeepSeek-V3 as the attacker LLM; Table 5 additionally compares LLaMA-4-17B, Qwen-Plus, and DeepSeek-V3. Each experiment is repeated five times. Arxiv and Products each sample 1000 target nodes; Cora uses the medium split, and the paper does not report the number of nodes in that split.
• Defense evaluation: Includes RUNG on Cora (Table 7, TF-IDF and SBERT) and adversarial training (Table 8). The text claims standard deviations are reported, but the tables do not provide standard deviation values.
• Evaluation details: Main results use node classification accuracy; Figure 4 uses ASR, and the paper does not provide its formula. The baseline side uses a GCN surrogate and BoW (A.4).

BadGraph is a black-box universal attack for node classification on text-attributed graphs, used to check whether the same topology plus text perturbation can drive both GNN reasoners and LLM reasoners toward predetermined incorrect labels.

• Problem: TAGs use both node text and topology. The authors argue that existing attacks that modify edges or features struggle with text, and embedding-space attacks are impractical; many PLMs are only accessible via API, so the attack must be black-box evasion.
• Approach: The graph encoder retrieves K=5 influencers by cosine dissimilarity. The LLM deletes one edge, adds one edge, and modifies the node text along the same influencer for each target node. The main experiment attacks DeepSeek-V3, with the authors claiming two queries per node.
• GNN numbers: In Table 1, for Arxiv, TAPE, GIN, Clean is 77.68 and BadGraph is 25.22; for Cora, SBERT, GraphSAGE, 86.00 vs. 39.93; for Products, TAPE, R-GCN, 89.54 vs. 80.52. The authors claim SOTA relative to the listed baselines and summarize the GNN scenarios in 6.2.1 as drops of nearly 40%, 52%, and about 30%.
• LLM numbers: In Table 2, for Cora, DeepSeek-V3 goes from 71.85 to 17.03, with the authors claiming a relative drop of 76.3%; for Products, DeepSeek-V3 is 77.10 vs. 55.80; for Arxiv, Mistral-7B is 9.30 vs. 2.90. The authors claim the joint perturbation is stronger than modifying only text or only structure.
• Authors' conclusions: The homophily upper bound and cross-modal shortcut theory are used to explain why small budgets can still change predictions and why this is not easily visible from global homophily. In Table 3, at a perturbation ratio of 0.17%, accuracy is 22.9% and edge homophily is 0.8182, compared to 0.8184 when unperturbed. The authors claim this is the first LLM-driven TAG attack and leave adaptive LLM agent attacks as future work.