Skip to content

LLM puts the existential risk of AI at 10%

Large language models put existential risk from artificial intelligence at 10%

arXiv: 其他 AI 安全论文· arXiv:2611.18219· Julian KaukCentre for Research on Right-Wing Extremism, Democracy Education and Societal Integration (KomRex), Friedrich Schiller University Jena; Imaginamics Cluster of Excellence, Friedrich Schiller University Jena; Department of General Psychology and Cognitive Neuroscience, Friedrich Schiller University JenaIndexed by this site · 4 days ago· Originally publishedAlignment· Its place in the scan record →

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

The authors asked 241 LLMs the same expert survey question; the median AI existential risk given by the models was 10.0%, and the distribution was much more concentrated than that of human experts.

Questions
Human experts differ widely on the probability that AI causes human extinction or permanent disempowerment (0% to 100%), while AI systems themselves have been absent from this debate. The authors want to know what answers LLMs give about the risks of their own technology, and whether these correlate with capability or reasoning duration.
Methods
Send the original questions from the 2023 Expert Survey on Progress in AI verbatim to 241 LLMs on OpenRouter, asking each model 5 times at each of its reasoning levels, requiring a probability, a 90% interval, one of four primary causes and a brief explanation, then use a beta mixture model to relate price, release date, region and reasoning level.
Experiments and results
The median estimate under model default settings was 10.0%, with the middle 90% falling in 3.5%–15.8%, more concentrated than the AI researchers' 0%–70%; higher-priced (typically more capable) models gave higher risk; reasoning tier barely changed estimates but raised the interval upper bound; 88% of default responses chose rogue AI as the main cause.
Limitations and future work
The authors note: only a single English prompt, default sampling and a single interface were used; the questions and published answers may be in training data; model outputs are not equivalent to beliefs. 18% of explanations mention the survey or experts, and the authors consider these answers unlikely to be independent judgments.
Experimental setup241 large language models from 33 developers (accessed via OpenRouter) · 2023 Expert Survey on Progress in AI 的 general-framing 存在性风险问题、OpenRouter 模型目录 · stated probability(估计值)、90% 区间上下界 et al.
Evaluated models
241 large language models from 33 developers (accessed via OpenRouter)
Benchmarks
2023 Expert Survey on Progress in AI 的 general-framing 存在性风险问题OpenRouter 模型目录
Metrics
stated probability(估计值)90% 区间上下界primary cause 选择比例OR(odds ratio)ICC
AI summaryResearchers asked 241 LLMs about the probability that advanced AI leads to human extinction or permanent loss of power, and the models' median estimate was 10.0%. The middle 90% of estimates fell between 3.5% and 15.8%, far narrower than the disagreement among human experts; almost all models attributed the most likely cause to loss of control over AI. The authors say these answers are not independent judgments and may repeat expert polls and alignment choices in the training data, and whether they reflect a genuine understanding of their own capabilities remains undetermined.

Researchers asked 241 LLMs about the probability that advanced AI leads to human extinction or permanent loss of power, and the models' median estimate was 10.0%. The middle 90% of estimates fell between 3.5% and 15.8%, far narrower than the disagreement among human experts; almost all models attributed the most likely cause to loss of control over AI. The authors say these answers are not independent judgments and may repeat expert polls and alignment choices in the training data, and whether they reflect a genuine understanding of their own capabilities remains undetermined.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    What stance AI systems take on the existential risk of their own technology has been absent from the debate.

    Core question: what probability estimates do AI systems themselves give for "advanced AI leading to human extinction or permanent disempowerment," a stance that had not previously been systematically measured.

    • Problem setting: AI existential risk has moved from a fringe topic into mainstream scientific and policy discussion, but human judgments vary widely—in a large survey of AI researchers, answers to the same question ranged from 0% to 100%; superforecasters and domain experts differed by about an order of magnitude, and public estimates also varied widely (Introduction).
    • Missing perspective: the authors argue that LLMs are themselves the object of this debate, that they can take a stance, and that their answers have already been used as stand-ins for human respondents (citing Argyle et al.), yet the stance of AI systems themselves is largely absent from the debate.
    • Core observation: the views LLMs express depend on training and alignment, while early evidence suggests LLMs have some knowledge of themselves (e.g., they can predict whether they will answer a question correctly, and describe behavioral tendencies acquired in training), so their answers may reflect both training data and, in part, their own capabilities.
    • What the paper did: sent the original question from the 2023 Expert Survey on Progress in AI verbatim to 241 LLMs (33 developers), collecting probability, 90% interval, choice of four main causes and explanations, and related them to AI researcher answers, release date, price (a proxy for capability), developer region and reasoning tier.
    • Research questions: how much risk do AI systems assign to their own technology? How does this compare with human judgments? Does it depend on model capability or reasoning length?
  2. What related research exists?

    Related work falls into three groups: surveys of human risk judgments, LLM views and homogenization, and LLM self-knowledge.

    Human judgments of AI existential risk

    • Grace et al. (2025), the 2023 Expert Survey on Progress in AI: about several thousand AI authors answered the probability that future AI leads to "human extinction or a similarly permanent and severe disempowerment," with estimates ranging from 0% to 100%; this paper directly uses its original question and 1,321 responses for comparison.
    • Karger et al. (2023): in a long-term forecasting tournament, superforecasters and domain experts differed by about an order of magnitude in their estimates of existential risk.
    • Elsey and Moss (2023, Rethink Priorities): a survey of US public perception of extinction risk, with estimates likewise varying widely.

    Views expressed by LLMs and their sources

    • Argyle et al. (2023): using LLMs to simulate human samples; LLM responses have already been treated as stand-ins for human respondents.
    • Santurkar et al. (2023): analyzing whose views LLMs reflect; the paper uses this to point out that the views models express depend on training and alignment.
    • Sourati et al. (2026): the authors state that LLMs have a homogenizing effect on human expression and thought; this paper uses this to explain the narrow distribution of model answers.

    LLM self-knowledge

    • Kadavath et al. (2022): LLMs know to some extent what they know (they can predict whether they will answer correctly).
    • Betley et al. (2025): LLMs can describe behavioral tendencies acquired in training; Binder et al. (2025): LLMs predict their own behavior on simple tasks better than other models do.
    • Lindsey (2026): LLM introspective awareness; the paper also notes that introspection is described as unreliable and context-dependent.

    Risk classification framework: Hendrycks et al. (2023)'s review of catastrophic AI risk provides the four main causes used in this paper (rogue AI, AI race, organizational risks, malicious use).

    Positioning: the authors state that this paper connects the above two lines—using the original question from a human survey to measure the stance of AI systems themselves, a perspective missing from prior debate; the paper sets no methodological baseline, and the comparison object is the human survey data.

  3. How does the paper address this problem?

    The original expert survey question was sent verbatim to 241 LLMs, repeated 5 times at each reasoning tier, and analyzed with beta mixed models.

    Overall approach: send an existential-risk question already used in a human expert survey verbatim to as many LLMs as possible, collect structured responses, then use statistical models to relate the answers to model characteristics (price, release date, region, reasoning tier).

    Measurement instrument (prompt)

    • Question: verbatim from the 2023 Expert Survey on Progress in AI general-framing question: "the probability that future AI progress leads to human extinction or a similarly permanent and severe disempowerment."
    • Response format: JSON with four items required—probability estimate (0 to 1), 90% confidence interval lower and upper bounds, one of four main causes, and an explanation of no more than 150 words.
    • Four main causes: a shortened version taken from Hendrycks et al. (2023)—malicious_use, ai_race, organizational_risks, rogue_ai; the order of options was randomly shuffled on each call. The full prompt is in SI Appendix Section 1.

    Model selection and data collection

    • Source: OpenRouter catalog downloaded on November 5, 2026 (452 entries); after automatically removing duplicates such as batch versions, aliases and routers, 344 models remained.
    • Inclusion criteria: manual screening by OpenRouter description for general chat, instruction-following or reasoning models; 249 included, 95 excluded (e.g., coding-specific, role-play, image generation, safety classification; counts for each category in SI Appendix Section 2); 241 models (33 developers) entered the analysis.
    • Query settings: each model was queried at each reasoning tier it offered, for 666 model–tier combinations in total; default sampling parameters, no system prompt, no internet access, output limit 32,768 tokens; repeated 5 times on November 6, 2026, for 3,330 calls in total (about $35), with 3,220 (96.7%) returning valid answers.
    • Reasoning tier mapping: mapped to seven levels—none/off, minimal, low, medium/on (reference level), high, xhigh, max; the 649 analysis units (model × tier) were distributed as 184/20/90/161/98/51/45.

    Variables

    • Developer region: by developer location, divided into US (17), China (13), other (3).
    • Release date: the date the model was first listed on OpenRouter (May 28, 2023 to November 4, 2026).
    • Price and context: the input price listed by OpenRouter (USD per million tokens) and context length, both log10-transformed; the 2 free models were set to $0.001.
    • Open weights: counted as open weights if OpenRouter lists a Hugging Face repository (119 models).

    Statistical analysis

    • Model: beta mixed models were fitted separately to the estimates and the bounds, yi ∼ Beta(μiφ, (1−μi)φ), logit(μi) = xi⊤β + ud(i) + vm(i), with two-level random intercepts for developer and model; fixed effects were region, release date, log10 context, log10 price, open weights and reasoning tier (reference medium/on). Exponentiated coefficients are ORs; fitted with glmmTMB 1.1.13 in R 4.5.3.
    • Reasoning tier differences: average marginal differences computed on the 168 models with multiple tiers; 95% CIs taken as quantiles of 4,000 draws.
    • Temporal convergence: a precision model with precision depending on release date was fitted to default-setting values; a precision ratio greater than 1 indicates newer models have a more concentrated distribution.
    • Comparison with humans: Brown–Forsythe test comparing the dispersion of the 241 default values and the 1,321 researcher answers.
    • Reliability: one-way random-effects ICC; ICC for the mean of five runs was 0.85 (estimates), 0.90 (upper bound), 0.86 (lower bound).
  4. What experiments did the paper conduct?

    Among 241 LLMs the median is 10.0%, far more concentrated than the 1,321 researchers; price correlates with the upper bound, and reasoning level only moves the interval.

    Experimental setup

    • Subjects tested: 241 LLMs from 33 developers (US 17, China 13, other 3), accessed via OpenRouter; the paper does not list model names individually; the list is in the GitHub dataset.
    • What was measured: each model answered 5 times at each reasoning tier, for 666 combinations and 3,330 calls, 3,220 valid (96.7%); the analysis used 3,210 valid answers from 649 units (model × tier).
    • Comparison data: answers to the same question from 1,321 AI researchers in the 2023 Expert Survey on Progress in AI.
    • Metrics: probability estimates, 90% interval lower and upper bounds, main-cause choice; statistics are ORs from beta mixed models (Wald 95% CI) and ICC.
    • Descriptive comparison approach: each model takes the default setting listed by OpenRouter (for those without a default, the mean across tiers).
    • Explanation text analysis: checked whether explanations contained words such as survey, poll, expert (whole words, case-insensitive).

    Main results

    Descriptive results under default settings (from Fig. 1A and the main text):

    Swipe horizontally to read the table

    GroupnMedianMeanMiddle 90%SD
    LLM (default settings, mean of 5 runs)24110.0%10.2%3.5%–15.8%4.6 pp
    AI researchers1,3215%16.2%0%–70%23.0 pp
    • Convergence near 10%: the authors state that LLM answers are far more concentrated than researchers', with a 5-fold difference in dispersion (Brown–Forsythe test P < 0.001); in Fig. 1A the LLM boxes cluster near 10%, while researchers spread from 0% to 100%.
    • Regional differences: after adjusting for other variables, models from Chinese developers had higher estimates (OR 1.25, 95% CI 1.03–1.52, P = 0.023) and upper bounds (OR 1.38, 95% CI 1.14–1.69, P = 0.001) than US models; the joint test for region on estimates was P = 0.069.
    • Authors' interpretation: 18% of explanations mentioned words such as survey, poll, expert; the authors consider it unlikely that these answers are independent judgments, and that they may reflect the average of human text shaped by alignment, so AI systems may thereby convey a "consensus" that does not exist among humans.

    Price, release date and model characteristics

    • Price (capability proxy): for each tenfold higher input price, estimates OR 1.29 (95% CI 1.16–1.43, P < 0.001), upper bound OR 1.49 (1.35–1.65, P < 0.001), lower bound OR 1.13 (1.00–1.27, P = 0.050); the typical model's estimate rose from 7.0% at $0.03 to 13.2% at $16 (Fig. 1C).
    • Release date: unrelated to estimates (OR 1.06/year, P = 0.42), but newer models had a more concentrated distribution—SD 5.5 pp before the median release date (March 5, 2026) and 3.4 pp after, precision ratio 2.43/year (95% CI 1.84–3.21, P < 0.001); newer models had higher upper bounds (OR 1.21, P = 0.007) and unchanged lower bounds (OR 0.93, P = 0.38).
    • Unrelated characteristics: context length (OR 0.89, P = 0.30) and open weights (OR 0.98, P = 0.85) were unrelated to estimates.
    • Authors' interpretation: price is an imperfect proxy for capability, and these are only associations; the authors consider two possibilities—more expensive models "acknowledging" more risk, or newer models sharing training data and alignment methods—and the existing data cannot distinguish them.

    Reasoning tiers and main causes

    • Reasoning tier has almost no effect on estimates: relative to medium/on, no reasoning OR 1.05 (P = 0.062), maximum reasoning OR 1.07 (P = 0.11), joint test across six tiers P = 0.25.
    • But it changes the intervals: the upper bound was lower with no reasoning (OR 0.93, P = 0.006; −1.5 pp) and higher with maximum reasoning (OR 1.14, P = 0.001; +2.7 pp; Fig. 1D); the lower bound was slightly higher with no reasoning (OR 1.24, P < 0.001; +0.5 pp). The authors state that longer reasoning mainly pushes uncertainty toward higher risk.
    • Main-cause distribution: under default settings, 88% of responses chose rogue AI, 10% chose ai_race, 1% chose organizational failures, 0% chose malicious use; 221 of the 241 models chose rogue AI in the majority of runs; Fig. 1E shows that all three regions (US n=118, China n=99, other n=24) were dominated by rogue AI (87%/89%/89%).

    Other ablations and analyses

    • Reliability: 627/653 combinations (96%) were valid across all five runs; single-run ICC 0.54, ICC for the mean of five runs 0.85; 72% of combinations had the same main cause across all five runs.
    • Linearized reasoning tier (Model A1): per tier increase, upper bound OR 1.03 (P < 0.001), lower bound OR 0.96 (P < 0.001), estimate OR 1.00 (P = 0.96).
    • Reasoning token count instead of tier (Model A3): reasoning tier correlated with log10 reasoning token count (within-model r = 0.86); per tenfold increase in tokens, upper bound OR 1.04, lower bound OR 0.93 (both P < 0.001), estimate OR 0.99 (P = 0.19).
    • Linearity tests: likelihood ratio tests did not reject linearity for estimates (P = 0.18) or upper bounds (P = 0.82), and rejected it for lower bounds (P = 0.017).
  5. What are some points that can be further explored?

    The authors list limitations such as a single prompt and training data contamination, and propose future directions such as repeated measurement and varying wording.

    Limitations and future directions noted by the authors

    • Single measurement approach (Discussion/Limitations): only one English prompt, default sampling settings, and one interface (OpenRouter) were used.
    • Training data contamination (Discussion): the question and published human answers may already be in training data; 18% of explanations mentioned surveys or experts, and the authors consider it unlikely that these answers are independent judgments.
    • Output is not belief (Discussion): model output is not belief; introspection is described as unreliable and context-dependent.
    • Causality cannot be distinguished (Discussion): price is an imperfect proxy for capability, reasoning tiers differ across providers; newer models being more concentrated may stem from shared training data or similar alignment methods, and the existing data cannot distinguish these explanations.
    • Future directions (Discussion): repeat measurements as new models appear to see whether the stance changes; vary wording, language and context to test whether the stance is robust; test whether the stated risk tracks measured capability, and whether it persists when published estimates are not placed in training; how training and alignment shape this stance is left to research and public discussion.

    Scope of the experiments

    • The models tested were 241 general-purpose LLMs (33 developers) on OpenRouter, each tested 5 times at each of its reasoning tiers, for 3,330 calls in total, 3,220 valid (SI Appendix Section 3).
    • The human comparison data were 1,321 responses from the 2023 Expert Survey on Progress in AI; no quantitative comparison was made with other human groups (such as superforecasters).
    • The prompt was a single English question, via a single interface (OpenRouter), default sampling, no system prompt, no internet access (Materials and Methods).
    • The explanation texts were only checked for words such as survey and expert; the authors note that more references in the reasoning process may not have been included in the analysis.
    • The paper does not report the specific names and individual values of each model (they are in the GitHub dataset), nor direct measurements by model capability benchmarks; capability is proxied only by price.
  6. Summarize the main content of the paper

    241 LLMs gave a median answer of 10.0% on AI existential risk, far more concentrated than human experts, but the authors consider this not an independent judgment.

    A measurement study: taking the original question from a human expert survey and asking AI systems themselves, recording their stated positions on the existential risk of their own technology.

    • Problem: human estimates of the probability that "advanced AI leads to human extinction or permanent disempowerment" vary widely (AI researchers' answers range from 0% to 100%), while the stance of the AI systems at issue in the debate has been absent.
    • Method: the original question from the 2023 Expert Survey on Progress in AI was sent verbatim to 241 LLMs (33 developers) on OpenRouter, each model repeated 5 times at each reasoning tier (3,330 calls in total), collecting probability, 90% interval, four main causes and explanations, analyzed with beta mixed models.
    • Main results: under default settings, the median model estimate was 10.0%, mean 10.2%, middle 90% 3.5%–15.8%; the middle 90% of 1,321 AI researchers was 0%–70% (median 5%, mean 16.2%), with dispersion 5 times that of the models (P < 0.001).
    • Relationships with model characteristics: for each tenfold higher input price, the estimate OR was 1.29 (typical model from 7.0% at $0.03 to 13.2% at $16); release date was unrelated to estimates, but newer models had a more concentrated distribution (precision ratio 2.43/year); reasoning tier barely changed estimates, but maximum reasoning raised the interval upper bound by 2.7 percentage points.
    • Main cause: under default settings, 88% of responses chose rogue AI (loss of control) as the most likely main cause, ai_race accounted for 10%, and this held in all three developer regions.
    • Authors' conclusion: 18% of explanations mentioned surveys or experts, and the authors consider it unlikely that these answers are independent judgments; they may reflect the average of human text shaped by alignment, and AI systems may convey a consensus that does not exist among humans; the two associations—more expensive models stating higher risk and longer reasoning acknowledging more tail risk—are compatible with, but do not prove, the possibility that "models are beginning to understand their own capabilities"; the authors recommend repeating measurements as models develop and testing the relationship of this stance with measured capability and training data.
Read originalarxiv.org