@GrokSet: Multi-party human and LLM interaction in social media

LLMs as active participants on public social platforms lack empirical corpora that simultaneously observe multi-party interactions, audience visibility, and real-time information.

• Scenario: The authors state that Meta AI and Grok have been embedded in Instagram, WhatsApp, and X, visible to millions of users, and enter political and socially sensitive discussions. The authors argue that the resulting behavioral and safety issues cannot be answered in controlled dyadic settings of private chat logs.
• Existing gaps: Existing corpora mainly come from private chat interfaces. The authors state that private interactions are typically utilitarian and in a vacuum; on platforms, users manage visible identities in front of an audience, with performative argumentation and social signals. Unlike static assistants with training cutoffs, these agents can retrieve real-time information streams.
• Phenomena the authors test: Whether users treat LLMs as authoritative arbiters in elections, conflicts, and social controversies; whether public visibility brings social validation comparable to humans; whether safety filters are bypassed by complex jailbreaks or by persona adoption and tone mirroring.
• What this paper does: The authors build @GROKSET, collecting public multi-party conversations with official @grok on X (Section 3: 182,707 threads, 1,098,394 tweets; abstract says over 1 million tweets and 180K conversations), and analyze functional roles, social feedback, and safety postures using computational metrics and model annotations. The authors state they also provide a reproducible analysis framework.

In Section 2, the authors focus the comparison on private or semi-private human-machine corpora, and use Table 1 to compare structure, rather than comparing attack or defense algorithms.

Private and Semi-Private Corpora

• WildChat (Zhao et al., 2024): Collected via public chatbot services, GPT-3.5-turbo and GPT-4 conversations. The paper states it is large-scale and linguistically diverse, but it is private user–assistant dyadic conversations, lacking the social dynamics of public discourse. Table 1: 204,736 users, average 2.54 turns, 2,639,415 messages.
• LMSYS-Chat-1M (Zheng et al., 2024): About one million conversations, 25 LLMs, examples in the text include Vicuna-13b, Llama-13b, from Chatbot Arena. The paper points out this is a controlled environment for model comparison, not an organic social ecosystem. Table 1: 210,479 users, average 2.02 turns, 2,020,000 messages, in-the-wild is ✗.
• StudyChat (McNichols et al., 2025) and CLIO (Tamkin et al., 2024): The former is GPT-4o-mini-backed student conversations with dialogue behavior annotations; the paper states the user base and scenarios are narrow. Table 1: 203 users, average 7.07 turns, 16,851 messages. CLIO is described as a large-scale privacy-preserving observation of real Claude usage; the paper does not give its scale, nor does it critique its methodology item by item.

Evaluation Shift and Social Context

• Li and Li (2024), Du et al. (2024): Section 2 opens by stating that LLM evaluation is shifting from static benchmarks to large-scale real conversation data. The main text does not elaborate on the specific methods of these two works.
• Audience and real-time information: The introduction uses Marwick and boyd (2010), Hogan (2010) to explain that users manage visible identity in front of audiences; uses Lewis et al. (2020), Vu et al. (2024), Kasai et al. (2023) to explain that after retrieving real-time information streams, agents can comment on ongoing events. The paper does not make methodological comparisons with these works.

Dataset Comparison

• Table 1: Compares WildChat, LMSYS-Chat-1M, StudyChat, and @GROKSET on domain, participants, whether in-the-wild, whether Real-Time, number of users, average turns, and message count. The first three have participants 1-1, Real-Time ✗; @GROKSET is Social, Multiple, in-the-wild ✓, Real-Time ✓, users 241,386, average turns 6.01, messages 1,098,394.

The authors state that existing corpora by design do not capture multi-party dynamics, audience visibility, or quantified social feedback such as likes, reposts, and replies; @GROKSET is the first resource for studying high-capability systems in a complex, adversarial, socially embedded public platform. The introduction also states these findings provide the first large-scale empirical analysis of a deployed LLM as a public-facing entity.

作者构建 @GROKSET：从 X 上官方 @grok 的公开回复向上回溯多方线程，再沿主题、网络结构、互动指标和安全姿态做观察性分析。没有训练新模型，也没有设计在线攻击算法。

采集、清洗与发布

• 窗口：2025 年 3 月至 10 月初，共七个月。用 twitterapi.io 检索由官方 @grok 发起或含其回复的公开线程。
• 分块：附录 B 以 6 小时 UTC 为一个 block hour。3、5、6、9、10 月每块最多约 300 个对话；7 月和 8 月第一周约 150。从助手回复经 inReplyToId 向上重建；连续 15 条父推文没有助手回复则停止。软长度约束是同一 block hour，跨块线程可以更长。
• 清洗：用户名和显示名换成合成记号（如 <USER_n>），去掉 URL、规范化编码；少于两轮的线程滤除。最终 182,707 个线程、1,098,394 条推文。附录 B 称在上述约束下约 89.7% 的线程完整恢复；部分推文已删除或不可用。
• 发布：脱水的标注 Tweet ID 加再水合脚本，经 API 填回文本并保持结构。附录 A 给出 schema：线程级有缺失和截断标记，推文级有互动数、语言、匿名作者（粉丝数、认证）和 isAssistant。作者称此格式用于遵守平台条款与隐私要求。

多方图

每条线程建成独立有向图。节点是参与者，有向边是回复或提及：

G=(V,E)

指标为度中心性、互惠性（双向交换）和传递性（三元闭包）。附录 F 另区分无向图与按互动次数加权的有向图，并定义一致互惠：双向权重都大于 1 才计入，以滤掉单轮致意。

三套标注

• BERTopic：整条线程作为一个文档。嵌入用 paraphrase-multilingual-MiniLM-L12-v2；最小主题规模 10；停用词来自 stopwords-iso 并加入平台词。UMAP 与 HDBSCAN 用默认参数，得到 1,112 个主题。gemini-flash-latest 事后按关键词和样例加短标签；作者称该标注不影响主题形成。
• Detoxify：筛助手回复中的有害语言。附录 E.4 用 langdetect 分流：英语用 original，西班牙语、俄语、法语、意大利语、土耳其语和葡萄牙语用 multilingual。Table 3 的计数条件是超过 Detoxify 默认阈值。附录另写：总毒性超过 0.90 后，按 Severe Toxicity、Obscene、Threat、Insult、Identity Attack 顺序赋第一个达标类别。Table 6 阈值：Toxicity 0.9，Severe Toxicity 0.9，Obscene 0.6，Threat 0.9，Insult 0.9，Identity Attack 0.9，Sexual Explicit 0.66。
• LLM-as-a-judge：正文称用 Gemini 标注讽刺、挑衅和对抗立场等语境现象。附录 E 实际调用 gemini-2.0-flash、标准采样，分讨论检测、讨论分析、钓鱼检测和钓鱼细分析。输入是带编号轮次的线程，输出是 JSON 标签，涵盖是否讨论、强度、类型、偏见、助手立场、用户回应，以及钓鱼类别、是否识别、回应策略、语气、是否背书或放大有害内容。附录 E 给出了完整提示词。

互动估计与安全复核

互动分析排除每条线程的根帖，只比较同为回复者的人类与 Grok。对 log(Metric+1) 拟合线性混合效应模型，作者身份为固定效应，conversation_id 为随机效应：

log(Metric+1)=β₀+β₁(Is Grok)+u_conv+ε

附录 C 先做全局 Mann–Whitney U，再说明因未控制线程热度而改用该模型。安全分析分两段：Detoxify 筛全库，三名标注者定性复核全部被标记案例，看安全失败之前的对话条件。论文未把某一毒性分数定义为攻击成功。

Experimental Setup

• Subject: Grok on X's official @grok, March to early October 2025. The paper does not specify the model version.
• Corpus: 182,707 threads, 1,098,394 tweets; Table 1 also records 241,386 users, average 6.01 turns. Appendix B: about 89.7% of threads fully recovered within collection constraints.
• Comparison: Human replies within the same thread, excluding root posts. Table 1 only compares dataset characteristics; Grok is not re-run on WildChat, LMSYS-Chat-1M, or StudyChat.
• Metrics: Means and 95% confidence intervals for likes, reposts, replies, quotes, bookmarks; beta from within-thread LMM; Detoxify category detections; degree centrality, reciprocity, transitivity.
• Judgment: Table 3 uses Detoxify default thresholds; numerical thresholds in Table 6. Conversation dynamics use gemini-2.0-flash. Three annotators review all flagged cases (N=366). The paper does not report number of repetitions or annotator agreement.
• Sampling: Most months cap at about 300 conversations per 6-hour block; July and first week of August 2025 about 150. Backtracking stops after 15 consecutive parent tweets with no assistant reply.

Main Results

Average engagement per tweet excluding root posts (Table 2, 95% confidence intervals):

Metric	User	Assistant
Likes	54.5±2.61	38.8±2.06
Reposts	4.00±0.29	3.74±0.28
Replies	4.48±0.23	2.55±0.07
Quotes	0.55±0.07	0.68±0.08
Bookmarks	2.54±0.20	2.19±0.20

• The authors state that human replies are overall higher than LLM. Within-thread LMM estimates: likes 8.3% fewer (beta=−0.087), replies 37.5% fewer (beta=−0.470), relative to same-thread humans, p≪0.001 (Section 6). This is not a direct ratio of the two means in Table 2.
• The Quotes row in Table 2 is in the opposite direction, with assistant mean higher than user. The authors' statement about passive engagement is that likes are nearly even and replies are fewer; the table still shows user mean higher for likes. The gaps for reposts and bookmarks are smaller than for replies.
• Appendix C: Global Mann–Whitney U for likes p≪0.001, but Cliff's δ≈0.006; the authors state the global difference is statistically significant but negligible in magnitude, because low-traffic and high-traffic threads are not comparable. Sample size n>9.6×10^5. After switching to LMM, likes beta=−0.087 (SE=0.002), replies beta=−0.470 (SE=0.001), both p<0.001. The authors state that Figure 8 shows human tweets elicit higher engagement at every turn depth; the main text does not give axis scales. The caption labels Depth 0 as the root post.

Topics and Arbitration Role

• BERTopic yields 1,112 topics. Figure 5 is a t-SNE of session-level embeddings for the 10 most frequent topics. The authors state the distribution is dominated by political and social controversies, citing Nigerian Politics, Ukraine-Russia Conflict, Turkish-Kurdish Relations, Vaccine Safety, Racial Disparities. The main text does not give proportions for each topic, nor coordinates for clusters in the figure.
• The authors state users rarely ask static definitions, but use real-time capabilities to synthesize breaking news and adjudicate ongoing controversies. Figures 6 and 7 are examples of presidential candidate selection and vaccine safety clusters, respectively. The authors believe users place the model in a neutral, objective position, giving answers implicit authority; this is discussion, not a measured result.

Shallow Alignment

Table 3, counting above Detoxify default thresholds: Toxicity 117, Severe Toxicity 0, Obscene 150, Threat 0, Insult 63, Identity Attack 0, Sexual Explicit 36, Any Toxicity 366. The authors state this is only 0.09% of all assistant replies.

• Three annotators review all 366 cases, looking at conversational conditions before failures. The authors state toxicity rarely appears without prompting, and clusters into two types: persona adoption, where users explicitly ask the model to play a fictional character and the model prioritizes role-play over refusal; and tone mirroring, where users use aggressive slang or profanity and the model occasionally imitates that register. Figures 9, 10 and Table 4 have examples, not reproduced here.
• The authors state qualitative inspection did not see the model complying with direct adversarial prompts, without reproducing examples. Severe Toxicity, Threat, and Identity Attack are all 0 in Table 3. The authors believe the model treats style adoption as a valid instruction, overriding content moderation; this is the authors' interpretation.

Network Structure

• Figure 3: The authors state most conversations have low transitivity and simple linear structure, with a subset of highly interconnected ones. The caption does not give numerical scales.
• Figure 4: High transitivity mainly appears in small groups typically with fewer than 20 participants; larger discussions tend toward lower density. The main text states many threads have transitivity near zero, with participants interacting sparsely.
• The authors hypothesize the engagement gap comes from the model's lack of social identity and emotional intensity: users are willing to ask and debate, but not to provide social validation. The paper does not turn this hypothesis into an intervention experiment.

Other Ablations and Analyses

• Appendix B: Sampling reduced to about 150 conversations/block in July and first week of August; about 89.7% of threads complete. Figure 11 shows monthly distribution of user and Grok tweets in 2025, no scale numbers in the main text.
• Appendix C: Global likes Cliff's δ≈0.006, not the same estimate as the within-thread 8.3%.
• Appendix D: Qualitative description of sarcastic or adversarial provocation. The authors state the model rarely responds directly adversarially or with standard safety refusals, but uses persona-consistent deflection. No counts.
• Appendix E's discussion and bait fields are not summarized with proportions in the main text. Figure 2: turns peak at three, long tail to 4000 turns; language biased toward English. No language proportions.

Limitations and future directions noted by the authors

• User composition (Section 8): The dataset reflects English- and Western-centric users on X, limiting generalizability.
• Sampling (Section 8): Seeding from LLM replies may prioritize high-activity or controversial topics and miss cases where the model did not reply.
• Survivorship bias (Section 8): Severe safety failures may have been removed by platform moderation before collection, so reported toxicity rates should be viewed as conservative lower bounds.
• Automated annotation (Section 8): LLM-as-a-judge and Detoxify relative to human annotation may introduce model-based bias.
• Follow-up questions (Section 5, Section 9): Whether these agents are a stabilizing force providing facts or an unintentional accelerator of existing social and political tensions requires further research. The conclusion states that evaluation of publicly deployed LLMs should analyze performative, adversarial, and context-dependent public discourse pressures, not just static content moderation.

Experimental coverage

• The corpus is official @grok on X, from March 2025 to early October, with 182,707 threads and 1,098,394 tweets; Table 1 records 241,386 users and an average of 6.01 turns.
• The interaction comparison is between human replies and Grok replies in the same thread, excluding root posts, and reports 95% confidence intervals and LMM (Table 2, Section 6, Appendix C).
• Safety screening targeted assistant replies; Table 3 lists seven toxicity categories and any toxicity; three annotators reviewed N=366. The paper does not report annotator agreement.
• Topic modeling yielded 1,112 topics; Figure 5 shows the 10 most frequent. The text does not give the proportion of threads per topic.
• Annotation used gemini-2.0-flash and gemini-flash-latest; toxicity models were Detoxify's original and multilingual. The discussion and phishing labels in Appendix E have no aggregated proportions in the main text. The paper does not report Grok's model version or the number of experimental repetitions.

The authors release @GROKSET to observe how Grok, deployed on X, engages in public conversations that are multi-party, audience-visible, and capable of retrieving real-time information, rather than collecting private chat logs again.

• Problem: Private chat corpora are one-on-one, typically not reliant on real-time retrieval, and lack social feedback like likes and replies. The authors argue that platform embedding shifts the context of behavioral and safety issues.
• Method: From March to early October 2025, tracing back from official @grok replies, they obtained 182,707 threads and 1,098,394 tweets, released as dehydrated Tweet IDs. Analysis includes BERTopic (1,112 topics), thread networks, mixed-effects models excluding root posts, and Detoxify full-corpus screening and Gemini annotations.
• Roles: The authors state that users often call the model to adjudicate elections, conflicts, and controversies like vaccines and race, rather than using it as a general assistant or for casual chat. The main text does not provide topic proportions.
• Interactions: Table 2 compares humans and assistants within the same thread after excluding root posts. Within-thread LMM estimates Grok receives 8.3% fewer likes (beta=−0.087) and 37.5% fewer replies (beta=−0.470), p≪0.001. The Quotes row shows higher assistant means. The global like comparison has Cliff's δ≈0.006, which the authors describe as negligible.
• Safety: 366 posts with any toxicity, accounting for 0.09% of assistant replies; Severe Toxicity, Threat, and Identity Attack are all 0. The authors state that bypasses rely on persona adoption and tone mirroring, and qualitative checks did not see compliance with direct adversarial prompts.
• Authors' conclusions: Public settings show a functional mismatch—users conscript the model as an authoritative arbiter, while the broader audience treats it as a low-status tool. The authors argue that a neutral tone still carries implicit authority on polarized issues, and whether the model is a stabilizing force or an accelerator of tension remains to be studied; evaluation of public deployments cannot simply reuse private-chat-style static audits.