Study: Large language model version changes weaken screening of AI-assisted scientific writing

Can detectors evaluated on a fixed version still screen scientific abstracts after LLM version updates?

• Scenario: Journals and conferences use automated detectors to estimate how much of a submission was written or revised by an LLM, and classify based on a score threshold. The authors state that the two types of errors fall on different people: when a human-written paper is classified as involving an LLM, the authors appeal; when undisclosed and disallowed LLM writing is classified as human writing, the burden falls on reviewers and editors.
• Existing limitations: The authors state that screening relies on benchmarks built from versions available at evaluation time; detectors degrade on LLMs not in the training data, including later versions of the same family. In scientific literature, the active use period of released LLMs shortens with release year, while retraining in service has costs, and false labeling of human text must be rare.
• What this paper does: It pairs 4,000 pre-ChatGPT PNAS abstracts with rewrites from 23 versions across 3 vendors, from June 2023 to August 2026, trains detectors under scenarios from retraining at every new version to training once and never updating, and compares two multi-version screening approaches and one deployed commercial detector.
• Authors' claims: The authors state that screening performance is not a fixed property of a detector and must be re-established as versions change. The policy recommendation in the abstract is to treat benchmark accuracy as a provisional result and re-validate at every LLM release, including earlier versions.

Use and screening in scientific writing

• Scale of use: Liang et al. (2025) estimate that by September 2024, LLM-modified content in computer science papers reached up to 22%; Kobak et al. (2025) report at least 13.5% of biomedical abstracts in 2024. The authors use these estimates to argue that use spreads faster than disclosure.
• Deployed screening: A computer science conference used a commercial detector to analyze all submissions to one track and rejected 178 papers in 2026 without review, accounting for 18.4% of that track; a sister journal of PNAS lists the same product among its evaluation tools.
• Distribution of mislabels: Liang et al. (2023) report that GPT detectors mislabel human text from non-native English writers at higher rates; Park et al. (2026) report that professional editing changes mislabeling rates. The authors say screening with low rates of mislabeled human text has been recommended as a priority.

Detection benchmarks and unseen models

• Fixed model pools: A set of benchmarks the authors discuss compare human text with generated or rewritten text using LLMs available at construction time; some also score on held-out generators, but the model pool stops at the time the benchmark was built, with no release order in which versions replace one another.
• Successive versions within a family: Work cited by the authors says detectors lose accuracy on LLMs absent from training data, including later versions of the same family. EvoBench evaluates detectors on successive versions of the same family, with supervised detectors trained only once on each family's earliest version.
• Cross-model sentence detection: Another recent work rewrites arXiv abstracts with multiple LLMs and performs sentence-level classification at a score threshold of 0.5, measuring how a detector trained on one model performs on other models.

Updating detectors and vocabulary signals

• Adapting to new generators: The authors mention training-free adaptation, class-incremental learning, lightweight transfer, and test-time adaptation, and say these methods address the cost of updating. This paper additionally asks which arriving versions require updates.
• Transfer and model properties: Prior research links cross-LLM detection transfer to parameter scale or the type of update that produced the model. The authors note that the scale and architecture of major versions are often not public. Embedding distances between text generated by different LLMs have also been reported to correlate with cross-model detection performance.
• Word distributions: Work cited by the authors says word-level distributions differ across LLMs and uses Jensen–Shannon divergence (JSD) to measure vocabulary distance between rewrites across versions. The authors treat this as a signal readable from the new version's text itself.

Baselines and data: The main detector is RoBERTa-base fine-tuned on a self-built PNAS corpus. Comparisons include four public detectors not trained on that corpus, a word-frequency detector, and the commercial detector Pangram. The data are 2009–2018 PNAS abstracts and their rewrites across 23 versions, not a replication of the external benchmarks above.

The authors' positioning is: retrain at each release, give the detector only versions released up to that point, and read results at a fixed mislabeling rate for human abstracts. The authors say detection loss concentrates in a few transitions rather than growing with model age, a shape left open by the benchmarks above.

The overall approach is to build a rewriting corpus, fine-tune a RoBERTa-base detector, and then read true positive rates at a fixed false positive rate by maintenance and screening method. The authors call text produced by any LLM AI-generated text, and the form studied here AI-rewritten: rewriting existing human abstracts.

Human Corpus and Rewriting

• Human corpus: from English articles in PNAS 2009–2018 in the OpenAlex snapshot of March 30, 2026. The authors assume this decade ends four years before ChatGPT's public release, thus predating AI-assisted scientific writing. After cleaning, 34,925 articles, labeled by OpenAlex as Health Sciences, Life Sciences, Physical Sciences, Social Sciences. 1,000 per domain, 4,000 total, are used for rewriting; the remaining 30,925 are used only for threshold calibration, not training.
• 23 versions: 14 from OpenAI, 4 from Meta, 5 from Alibaba, released from June 2023 to August 2026, with identifiers and ordering dates in Table S1. The authors state that Anthropic's Claude was excluded due to vendor requirements. OpenAI includes each version of the GPT-4 to GPT-5 text model line within the window; parallel mini, nano, pro, o-series, and aliases that update in place are out of scope. Meta and Alibaba start from Llama 3.1 and Qwen2.5, taking the mainline instruction model at each generation at the size tier consistently provided across generations by the vendor.
• Two-stage rewriting: each version uses the same procedure to rewrite each abstract. The procedure comes from existing literature: first compress the original into bullet points, then expand into a paragraph in abstract genre. The authors add a segment in the second stage stating this is a PNAS article abstract and citing the journal's requirements on length and audience, and that the output should be only the abstract itself. The full text and the added part are marked in Supplementary Section S2.
• Sampling and cleaning: self-hosted 9 Meta and Alibaba versions, and 6 GPT-4 series API versions exposing the corresponding controls, at temperature 1.0, generation limit 2,048 tokens. The 8 GPT-5 series only accept default temperature 1.0, using provider default settings, limit 4,096 tokens, to cover tokens used for internal reasoning. Of 92,000 requests, 47 (0.05%) were discarded: 38 hit the limit, 9 were empty, none refused. 91,953 retained. Additionally, 349 had conversational openings or closings removed, without altering the abstract body.

Detector

• Model and split: the detector is a binary classifier labeling abstracts as human-written or AI-rewritten. Fine-tuned on 125M-parameter RoBERTa-base. Input is at most the first 512 tokens. Training and evaluation are limited to 3,955 papers that have rewrites retained across all 23 versions, split 80/20 by paper so that the human abstract and all rewrites of the same paper fall on the same side: 3,164 train, 791 eval. Each training paper uses the human abstract plus one AI rewrite, balancing the two classes.
• Optimization: 4 epochs, batch size 8, AdamW, learning rate 2×10^-5 with linear decay. Architecture, training procedure, and hyperparameters are fixed across experiments, except for the pooled detector in Supplementary Section S8.2 trained on all available rewrites. Each experiment is repeated 10 times with random splits, reporting mean and standard deviation.
• Why train in-domain: the authors first look at four public detectors not trained on this corpus, then train on this corpus. The authors state that this way the detector learns the same type of text it will later judge, a favorable setting for detection.
• Typography normalization: human abstracts, after publication and indexing, have typography flattened to plain text; rewrites retain LLM typography. The authors apply the same deterministic normalization to all text used for self-trained detectors, calibration, evaluation, and JSD, converting quotes, dashes, sub/superscripts, and markers to the plain-text form of human abstracts. Text submitted to commercial detectors, and rewrites seen by two readers, are pre-normalization text.

Four Maintenance and Two Screening

• Maintenance scenarios: when judging target version v, training versions are same-vendor as v. current version uses only v's rewrites. past and current versions uses v and earlier versions from that vendor. past versions only uses only earlier versions, corresponding to when new-version text arrives before training data for that version exists. frozen at a specific version trains only on the earliest version from that vendor in this study, never updating: GPT-4, Llama 3.1, or Qwen2.5. The three GPT-5.6 versions share the same date, added one by one to screening in alphabetical order by name; in maintenance scenarios, none counts as earlier than another.
• union: 23 specialized detectors trained per current version run in parallel, each with its own 1% false positive threshold, and if at least one triggers, the text is labeled AI-rewritten.
• pooled: one classifier trained on rewrites from all 23 versions. When pooling multiple versions, each training paper still draws only one AI rewrite, uniformly from the pooled versions, drawn once before training and held fixed across epochs. Thus each scenario has the same number of training papers, and pooled measures detection under a fixed training budget. pooled(k) takes the first k versions by release date, k from 2 to 23.
• Growth over time: sort the 23 versions by release date, and for k=1 to 23 build a union of the first k specialized detectors, and a pooled trained on the first k versions. k=23 is the two screenings above. The authors state Muse-Glimmer is the latest released among the 23 versions.

Vocabulary Distance

• Algorithm: each version pools its retained rewrites, lowercases, tokenizes by longest alphanumeric contiguous segments, drops tokens that are only digits, and obtains normalized relative word frequencies. Between version pairs, compute JSD: symmetrized KL divergence base 2, 0 if distributions identical, 1 if no common words.
• Comparison with detection: for 253 unordered version pairs, compare JSD with the average of TPR at 1% false positive in both training–test directions. The authors use Spearman rank correlation and evaluate with a Mantel permutation test shuffling the 23 version labels.
• Author-suggested use: the authors state JSD requires only a sample of LLM-rewritten text and can serve as a candidate input for deciding which newly released versions may need revalidation of in-service detectors.

Thresholds and Commercial Judgment

• Self-trained detectors: higher scores lean more toward AI-rewritten. Thresholds are computed only on the 30,925 calibration abstracts, which no detector saw during training. union and pooled use all 30,925; other experiments use a fixed random subsample of 5,000. The main operating point sets false positive rate at 1%, with threshold at the 99th percentile of calibration scores. The authors state this follows recommendations to evaluate AI-text detectors at a fixed low false positive rate. 5% and 0.5% are in the supplementary. Reported is the proportion of AI rewrites in evaluation papers whose scores reach the threshold, i.e., TPR.
• Commercial detector: Pangram version 3.3.2, vendor default model as of September 2026, queried via API under production settings, with no access to its training data. The authors note that after queries, the vendor decommissioned this version on September 30, 2026. It returns three document-level labels, and the authors do not choose thresholds. Misses count only rewrites labeled fully human-written, which the authors state is the most favorable counting for the detector; the other two labels count as correct detection. False positives are the proportion of the 5,000 calibration abstracts not labeled fully human-written.

实验设置

• 被改写的模型：OpenAI的GPT-4、GPT-4 Turbo、GPT-4o（2024年5月、8月、11月）、GPT-4.1、GPT-5、GPT-5.1、GPT-5.2、GPT-5.4、GPT-5.5、GPT-5.6 Luna、Sol、Terra；Meta的Llama 3.1、Llama 3.3、Llama 4 Maverick、Muse-Glimmer；Alibaba的Qwen2.5、Qwen3、Qwen3.5、Qwen3.6、Qwen3.8。共23个，Table S1。
• 语料规模：改写请求92,000次，保留91,953篇。检测主实验用其中3,955篇论文，10次论文级随机划分。校准摘要30,925篇；除union和pooled用全部外，其余实验用5,000篇子样本。
• 自训检测器：RoBERTa-base，4个epoch，batch size 8，AdamW，学习率2×10^-5。主指标为TPR at 1% FPR。Figure 1的点是10次划分的均值，误差棒是标准差。
• 公开检测器：未在本语料上训练。OpenAI的GPT-2输出检测器用roberta-base-openai-detector；RADAR用TrustSafeAI/RADAR-Vicuna-7B；Fast-DetectGPT的无采样分析版以EleutherAI/gpt-neo-2.7B同时做打分和参照；Binoculars用tiiuae/falcon-7b与tiiuae/falcon-7b-instruct。各读至多前512 token，阈值同样设在5,000篇校准分数的第99百分位。
• 商用检测器：Pangram 3.3.2。每个版本提交同一批400篇论文的改写，学科构成为Health Sciences 40、Life Sciences 85、Physical Sciences 185、Social Sciences 90，另加5,000篇校准摘要。
• 其他：词频检测器、0.5%与5%假阳性、两种单阶段替代提示词。成功判定是阈值或厂商标签，不是LLM裁判。

主结果

正文在Figure 1的四种维护场景下给出的TPR at 1% FPR如下。表中“18个版本”指20个有更早同厂商版本者里面、没有落入两处低点的那些。

维护场景	版本或汇总	TPR at 1% FPR
Current version	23个版本的中位数	99.9%
Past and current versions	23个版本的中位数	99.3%
Past versions only	上述18个版本的中位数	95.4%
Past versions only	GPT-5.1（这18个中最低）	85.5%
Past versions only	GPT-5	26.6%
Past versions only	Muse-Glimmer	3.8%
Frozen at GPT-4	GPT-5及之后任一版本	至多0.6%
Frozen at Qwen2.5	Qwen3.8	86%

• 两处低点：作者称只训更早版本时，20个有前驱的版本里有18个的TPR仍高于85%。两处低点分别是GPT-5代的第一个版本，以及Meta的Muse家族第一个开放权重模型。摘要把最陡边界写成之前高于99%、之后为3.8%；正文另给出Llama 4的mixture-of-experts过渡为99.2%，该版本是Muse-Glimmer在Meta序列中的前一个。
• 不是每次更换都掉：转到Qwen3架构时，past-only的TPR为94.7%。冻在各厂商最早版本上的轨迹因厂商而异，且不是单调下降。
• 作者的解读：作者称这些下降集中在少数版本过渡，而不是随模型年龄逐渐累积；并称这与检测器在训练数据未覆盖的LLM上变差的先前结果部分一致。

两种筛查场景

• 并排与汇集：union对Muse-Glimmer平均检到98.6%，其他版本更高。10次划分上，它误标11.9%±1.2%的留出人类摘要，即每划分791篇中检测器训练时没见过的原文。作者称大约八分之一。pooled误标0.8%±0.3%，接近设定的1%；23个版本的漏检率中位数为2.8%，Muse-Glimmer为33.8%±3.4%，其次Qwen2.5为8.8%，GPT-5.6 Terra为6.6%。作者称，训练数据里已有Muse-Glimmer改写，仍漏掉约三分之一。
• 按发布顺序累加：Figure 3中，union的人类误标从k=1的1.4%升到k=23的11.9%；Muse-Glimmer漏检在k=22为17.8%，k=23降到1.4%。pooled的人类误标在每个k都接近1%；k=12至22时Muse-Glimmer漏检高于50%，k=23为33.8%。只用GPT-5之前11个版本训练的pooled，检到GPT-5改写的26.7%，与只训OpenAI更早版本的26.6%接近。
• 作为单一筛查校准：Supplementary Section S8.1把union收成一个分数后再放到约1%假阳性。人类摘要误标为1.1%±0.4%，23个版本的中位TPR为99.3%。Table S7中Muse-Glimmer的TPR为83.6%±2.8%，对照pooled的66.2%±3.4%。作者称这样仍漏掉Muse-Glimmer改写的六分之一。

商用检测器Pangram

• 人类侧：5,000篇校准摘要中，4,999篇标为fully human-written，1篇标为fully AI-generated。假阳性率0.02%（1/5,000；精确95%区间0.0005%至0.11%）。作者称这远低于自训检测器所用的1%。
• 版本侧：漏检只计fully human-written。Figure 4(a)从GPT-5的6.5%（26/400）到Muse-Glimmer的79.8%（319/400）。与发布日期的Spearman ρ=0.25（p=0.25），作者称没有显著关系；GPT-5等较晚版本反而在较容易检到的一侧。
• 学科：跨全部AI改写摘要，漏检为Health Sciences 45.2%、Life Sciences 33.4%、Physical Sciences 31.4%、Social Sciences 26.5%。上述400篇的学科篇数并不相等。作者称漏检在学科间不均匀，并称人类文本的标记此前也被报告为分布不均。作者把Pangram和pooled放在同一侧：假阳性更少，漏检更多。

词表距离与跨版本迁移

• 格子：Figure 2(a)是行版本训练、列版本测试、10次划分平均的TPR，按全部23×23格的百分位着色，不是绝对色标。正文点名的格子：GPT-4o（2024年11月）训的检测器检到Llama 3.3改写的99.5%；GPT-4训的检到GPT-5.4的0.2%；GPT-5.6 Luna、Sol、Terra训的检到GPT-4至多0.2%。作者称格局跟着模型代际走，而不是跟着厂商走，并称这与检测可以在不同LLM之间迁移的先前结果部分一致。
• 相关：253个无序对上，JSD越大，两个方向TPR的平均越低，Spearman ρ=-0.85，p<0.001（Mantel置换，Supplementary Section S6）。同厂商ρ=-0.92，跨厂商ρ=-0.74。控制发布间隔、是否跨厂商、平均词数差之后，偏Spearman相关为-0.80。与最近同厂商前驱的JSD最远的两个版本，正是past-only TPR最低的Muse-Glimmer和GPT-5。
• 反向：同厂商、发布日期不同的104对里，41对是较新版本训的检测器检到较旧改写不到一半，47对是较旧训的检到较新不到一半。作者称JSD也跟踪这一反向。Discussion称，控制发布时间、厂商和改写长度后关联仍在，但这不构成因果。

其他消融与分析

• 词频检测器、只训更早版本、1% FPR：GPT-5为4.4%±0.6%，Muse-Glimmer为1.8%±0.4%；与JSD的Spearman ρ=-0.83（Fig. S3、S6的同一检验）。
• 同一批RoBERTa分数只改阈值：5% FPR下past-only的GPT-5为57.9%±7.8%、Muse-Glimmer为14.0%±4.3%；0.5% FPR下为19.0%±4.3%与2.2%±1.2%（Fig. S4、S5）。两处的JSD相关均为ρ=-0.85。
• 两种替代提示词、各约500篇、1% FPR：past-only的GPT-5为5.8%与7.9%，Muse-Glimmer为15.3%与7.1%（Table S6）。同规模的原两阶段提示为GPT-5的10.2%与Muse-Glimmer的2.6%；全文规模则是表中的26.6%与3.8%。
• pooled改用每个汇集版本的全部改写：k=12至22仍漏掉Muse-Glimmer的44%至67%，k=23为37.9%±25.2%（10次划分从10.7%到74.7%）。230个此类检测器中22个对训练版本的中位TPR低于50%；主设定一稿一改写的220个里，最低该类中位TPR为96.6%（S8.2）。
• 四个无领域内训练的公开检测器，23个版本的中位TPR：OpenAI检测器0.1%，RADAR 0.5%，Fast-DetectGPT 4.3%，Binoculars 2.5%。任一版本最高30.3%（Binoculars，Llama 4 Maverick）（Table S5）。
• 改写保真：91,953篇上，23个版本的嵌入余弦均值最小0.899（GPT-5）、最大0.946（Muse-Glimmer与Qwen2.5）；随机不同原文对的均值为0.109（Fig. S1）。人工阅读40篇×GPT-5与Llama 4 Maverick，不估计实质改动的频率（S4）。

Limitations and future directions noted by the authors

• Journal, language, and genre: The results come from one journal, one language, and one text type, namely English abstracts from PNAS 2009–2018. The authors state that the generality of the qualitative findings requires further study (Discussion).
• Assistance modality: LLM use was simulated as a two-stage rewriting of abstracts. The authors state that actual use ranges from grammar correction to full rewriting and may involve multiple rounds of interaction between authors and LLMs; systematic studies by degree and type of assistance are needed to see how far the behavior extends (Discussion).
• Detector types: The simulation relies on one class of trained detectors, namely RoBERTa-base fine-tuned on this corpus, and a commercial detector at a fixed operating point. A word-frequency detector reproduced the drop at the generation boundary and its association with JSD; four public detectors missed most rewrites for each version, but the authors state this covers only a few designs, and detectors based on other principles may behave differently (Discussion).
• Version range: The 23 versions come from three vendors, namely OpenAI's GPT-4 to GPT-5 text model line, and Meta and Alibaba versions after Llama 3.1 and Qwen2.5. The authors state that other vendors, earlier releases from these three, and lighter models released in parallel by the vendors were not included (Discussion).
• Not causal: The association between JSD and detection transfer persists after controlling for release time, vendor, and rewriting length. The authors state this does not constitute causation (Discussion).
• Whether it can serve as a signal in the future: The authors state that JSD is a candidate input for judging which newly released versions may need re-evaluation, and whether it is useful for future releases warrants further study (Discussion).

Experimental coverage

• Corpus: The human side is PNAS 2009–2018 English articles from the OpenAlex snapshot of March 30, 2026, with 34,925 articles after cleaning; 1,000 articles were sampled for rewriting from each of four OpenAlex subject areas; 3,955 articles were retained for rewriting across all 23 versions; the remaining 30,925 were used for calibration (Methods 4.1).
• Generation side: 14 OpenAI versions, 4 Meta versions, and 5 Alibaba versions, from June 2023 to August 2026; the authors state that Anthropic's Claude was excluded due to vendor requirements. The final availability check date was September 13, 2026 (Methods 4.2, Table S1).
• Detectors and repetitions: The self-trained detector is RoBERTa-base, with a primary operating point at 1% FPR, supplemented by 0.5% and 5%; 10 paper-level random splits per setting. The commercial side is Pangram 3.3.2, with the same batch of 400 rewritten abstracts per version plus 5,000 calibration abstracts, without self-selected thresholds (Methods 4.3–4.5).
• Comparison detectors: Additionally evaluated were a word-frequency detector, and OpenAI GPT-2 output detector, RADAR, Fast-DetectGPT, and Binoculars, none trained on this corpus (Supplementary Sections S5, S7).
• Rewriting verification: Four automated fidelity metrics covered all 91,953 retained rewrites; human reading covered 40 papers times GPT-5 and Llama 4 Maverick. The authors state that this verification characterizes recurring changes in the sample and does not estimate the frequency with which rewriting changes the substance of the original across the 23 versions (Supplementary Section S4).

The authors quantify how screening of AI rewriting of scientific abstracts changes when LLM versions keep changing and false labeling of human abstracts must be rare.

The problem arises because journals and conferences already screen submissions by score thresholds, while benchmarks mostly stop at versions available at evaluation time. Using 4,000 pre-ChatGPT PNAS abstracts, with two-stage rewriting by 23 versions from OpenAI, Meta, and Alibaba, they fine-tune RoBERTa-base and compare maintenance strategies at 1% false positives.

• When trained on the tested version itself, the median TPR across 23 versions was 99.9%; when retrained together with earlier versions, the median was 99.3% (Figure 1).
• When training only on earlier versions from the same vendor, GPT-5 was 26.6% and Muse-Glimmer 3.8%. Under the same method, switching to the Qwen3 architecture gave 94.7%, and switching to Llama 4's mixture-of-experts gave 99.2%. When frozen on GPT-4, any version from GPT-5 onward was at most 0.6%.
• Taking the union of 23 dedicated detectors falsely labeled 11.9%±1.2% of held-out human abstracts; pooling all versions into one detector missed 33.8%±3.4% of Muse-Glimmer rewrites, even though that version was in the training data.
• Pangram 3.3.2 labeled only 1 of 5,000 calibration abstracts with a label other than human, yet labeled 79.8% (319/400) of Muse-Glimmer rewrites as fully human-written.
• Across 253 version pairs, the Spearman ρ between vocabulary JSD and the bidirectional average TPR was -0.85 (p<0.001). Detectors trained on newer versions also miss older versions; the three newest OpenAI versions detected at most 0.2% of GPT-4 rewrites.

The authors state that screening performance must be re-established after each release, and re-validation should include earlier versions; which versions the detector continues to cover also determines whose text it is more likely to miss. The authors also state that the association between JSD and transfer success is not causal, and results come from PNAS English abstracts, two-stage rewriting, and the detector types tested.