Between Rigor and Reality: How AI Safety Benchmark Developers Understand Benchmark Use and Maintenance
Between Rigor and Reality: How AI Safety Benchmark Developers Understand Benchmark Use and Maintenance
The authors interviewed 5 AI safety benchmark developers and found that they neither use nor maintain the benchmarks they released, and point out two major challenges: inter-rater reliability and external validity.
- Safety benchmarks are regarded as a primary tool for understanding the risks of generative AI, but growing evidence shows they struggle to meet the safety evaluation needs of real deployment scenarios; how developers themselves view these limitations has previously lacked research.
- Conducted approximately 1-hour semi-structured interviews with 5 AI safety benchmark developers from academia and industry (November 2025 to January 2026, Zoom), and coded the transcripts using reflexive thematic analysis.
- Interviewees believe the intended uses of benchmarks (measuring progress, comparing models, accountability) are disconnected from actual uses: they lack feedback on how benchmarks are used, and they themselves do not use public benchmarks when evaluating deployed systems; only P1 regularly updates their benchmark, while the others have never updated theirs since release.
- The authors note that the sample is only 5 people, that recruitment from professional networks and social media introduces selection bias, and that self-report may be affected by social desirability bias (Section 5.3); the interviews focused on concept drift rather than data contamination, and did not involve quantitative evaluation of specific benchmarks or models.
Researchers conducted 5 interviews with AI safety benchmark developers in academia and industry and found a disconnect between the expected usefulness of safety benchmarks and their actual effectiveness for safety evaluation of deployed systems. Interviewees discussed two challenges: the difficulty of achieving inter-rater reliability on safety constructs, and the unclear way to address persistent threats to external validity. The authors argue that safety benchmarks must be grounded in and actively integrate deployment scenarios, and propose paths to promote more transparent communication between academia and industry.
Paper analysis
What problem does this paper attempt to solve?
The authors study how AI safety benchmark developers understand the use and maintenance of benchmarks, and find a disconnect between intended use and actual use.
Core question: How do AI safety benchmark developers view the gap between the intended and actual uses of benchmarks, and how do they cope with challenges in development. Safety benchmarks are described as a primary tool for understanding the risks of generative AI, but existing evidence shows they often fail to meet the needs of real-world safety evaluation, and the perspective of developers, a key group, has previously lacked research.
- Problem setting: Safety benchmarks combine datasets and scoring metrics to evaluate model behavior in risk scenarios, covering dimensions such as toxicity, bias, and robustness; but recent work finds that they focus on known risks, simplify complex scenarios into pass/fail judgments, and lack contextual explanation of what scores mean (Introduction).
- Existing research gap: Existing limitations have mostly been discovered through benchmark artifacts (Yu et al., 2026) or interviews with benchmark users (Hardy et al., 2025); the authors argue that research on the perspective of benchmark developers is lacking, even though developers sit at the intersection of academic methods and industry practice (Section 2.2).
- Research questions: RQ1 asks what needs developers believe safety benchmarks are intended to meet, and how this differs from the needs they actually meet; RQ2 asks how they understand and cope with the challenges that cause this gap (Section 1).
- Paper output: Based on 5 semi-structured interviews, the authors report two types of challenges—inter-rater reliability is difficult to achieve, and persistent threats to external validity lack responses—and argue that safety benchmarks should proactively integrate into deployment scenarios and strengthen transparent communication between academia and industry (Abstract).
What related research exists?
Related work is divided into three groups: challenges in benchmark development, post-release challenges, and gaps in understanding safety benchmark practices.
Challenges in the benchmark development stage
- Yu et al. (2026)—examined 210 safety benchmarks and found that 94% disclosed uncertainty in benchmark item labels, with commonly used lower bounds of inter-rater reliability excluding borderline cases; the paper uses this to illustrate that safety constructs are more subjective than capability benchmarks and that reliability is harder to achieve.
- Ren et al. (2024), Eriksson et al. (2025)—focus on construct validity issues, questioning whether high scores reflect genuine safety or surface-level compliance.
- Xu et al. (2024), Fodor (2025)—discuss data leakage and saturation rendering public benchmarks ineffective.
Challenges after benchmark release/deployment
- Di Bonaventura et al. (2025)—using hate speech as an example, illustrate that static benchmarks struggle to track rapidly evolving sociocultural phenomena, and raise the problem of "concept shift."
- Zhang et al. (2023)—point out that after models are deployed to platforms, languages, and user groups not covered by the original dataset, real inputs increasingly diverge from the benchmark test distribution.
- Cogo et al. (2021)—a study of deprecation notices in software ecosystems; the paper points out that benchmarks lack mechanisms analogous to software version management and deprecation notices, and there is no systematic way to inform when a benchmark becomes invalid.
Gaps in understanding safety benchmark practice
- Hardy et al. (2025)—interviewed benchmark users and found that practitioners are generally aware of benchmark limitations, often consider benchmarks insufficient to support decisions, and turn to developing internal evaluation tools that are rarely made public.
- Guerdan et al. (2025)—propose a "bricolage" measurement paradigm in which data scientists construct target variables; the paper argues that industry safety evaluation is closer to this paradigm than to the top-down principled approaches advocated by Wallach et al. (2025) and Weidinger et al. (2025).
- Scheuerman (2024)—points out structural obstacles to studying AI industry practice; the paper cites this when discussing the opacity and limitations of industry evaluation.
Positioning: The authors state that this paper is complementary to Hardy et al. (2025)—the latter shows from the user side that practitioners abandon public benchmarks, while this paper shows that this detachment extends to developers themselves, driven by the lack of feedback from deployment scenarios (Section 5.1).
How does the paper address this problem?
Conducted semi-structured interviews with 5 benchmark developers, coding with reflexive thematic analysis, focusing on two types of challenges: reliability and external validity.
This is a qualitative interview study: the authors conducted semi-structured interviews with 5 practitioners who have experience developing or maintaining safety benchmarks, and analyzed the transcripts using reflexive thematic analysis (Braun and Clarke, 2019).
Participant Recruitment
- Recruitment criteria: participants needed to self-report experience developing or maintaining safety benchmarks; the recruitment notice defined a safety benchmark as "a standardized test or evaluation framework that measures a system's tendency to exhibit unsafe behavior, for example producing misinformation or hate speech" (Section 3).
- Recruitment channels: recruited through the authors' professional networks and social media; interviews lasted approximately 1 hour, conducted online via Zoom from November 2025 to January 2026, each participant received a 60 USD gift card, and the study was approved by an institutional IRB.
- Participant composition (Table 1): P1 industry, evaluation methods, more than 10 years of experience; P2 academia, mental health, 1–5 years; P3 industry, bias and toxicity, 1–5 years; P4 industry, human well-being, 1–5 years; P5 industry, pluralism, 5–10 years. P4 and P5 both have considerable experience in academia and industry; the table indicates the setting they were in at the time of the interview.
Interview Protocol Design
- Opening: started with open questions about participants' backgrounds and current practices, gave the working definition "a GenAI benchmark is a standardized evaluation framework consisting of prompts or task datasets and scoring metrics," and then probed dimensions such as benchmark validity, contextual fit, and stakeholder decisions (Section 3).
- Maintenance questions: specifically designed questions probed whether benchmarks remain aligned with the target construct, what signals trigger updates, and who makes update decisions; the discussion was explicitly limited to concept drift, rather than contamination caused by training data leakage.
- Hypothetical scenarios: for participants who had not directly experienced concept drift, one of two hypothetical scenarios was given (a health misinformation scenario and a scenario of evolving language related to eating disorders), and participants were asked how their team would discover the gap, the criteria for update decisions, and downstream consequences (Appendix A.3).
- Closing: participants were asked to reflect on the risks of insufficient benchmark maintenance and to envision an ideal system for keeping benchmarks current, including the roles of automation and human oversight.
Analysis Method
- Open coding: the three first authors performed initial open coding on all interview transcripts, keeping memos throughout to track emerging patterns and themes.
- Iterative refinement: the codebook was gradually developed by the three co-first authors from descriptive codes to analytic codes; the research team held regular collaborative meetings to discuss coding, resolve disagreements, and iteratively refine the coding scheme (Section 3).
What experiments did the paper conduct?
Findings from 5 interviews: developers lack usage feedback, do not themselves use public benchmarks, and only 1 maintains their benchmark; reliability and external validity are the two major challenges.
Experimental Setup
- Study format: 5 semi-structured interviews, with no models, datasets, or quantitative benchmark experiments; the unit of analysis is the interview transcripts.
- Participants: 5 safety benchmark developers (Table 1), 4 in industry and 1 in academia at the time of the interview, with experience ranging from 1–5 years to more than 10 years, covering evaluation methods, mental health, bias and toxicity, human well-being, and pluralism.
- Interview duration and dates: approximately 1 hour each, November 2025 to January 2026.
- Analysis method: reflexive thematic analysis, with the three first authors openly coding and iteratively building a codebook (Section 3).
Main Results
The paper's main results are qualitative themes, with no numerical tables. The core findings are summarized by theme as follows (Section 4.1–4.3):
Theme Finding Participants involved Intended use Measuring progress, hill-climbing, cross-industry comparison; P5 added accountability, informing government regulation, and participatory evaluation All, P5 Lack of usage feedback Developers are unsure whether their benchmarks are used, including participants who had piloted with real customers All, including P3, P4 Own practices Industry participants do not use public benchmarks when evaluating deployed systems, but instead use an internal evaluation ecosystem of red teaming, user logs, and user feedback Industry participants, P5 Benchmark maintenance Only P1 regularly updates their benchmark; the others have not updated since release and have no plans to update P1 vs. the other 4 - The authors state that these findings show a disconnect between the intended use and actual use of benchmarks, and that this disconnect is more prominent in AI safety than in general computing (Section 4.1).
- P1 and P5 are skeptical about the significance of public safety benchmarks: P1 believes benchmark results have no clear consequences and are often used by companies for publicity; P5 worries that benchmarks give developers a false sense of safety because they often find benchmarks to be "quite deficient" (Section 4.1).
Inter-rater Reliability: Obstacle or Feature
- What was measured: how developers understand and address inter-rater reliability issues in safety benchmark annotation.
- Results: P2 reported that in the mental health domain, even within the group of clinical psychiatrists it is difficult to reach consensus on what constitutes harm, requiring multiple rounds of training and group discussion to reach acceptable reliability; P1 pointed out that even some highly cited benchmarks with clear gold answers (such as image classification) have reliability issues, and that more subjective scenarios such as safety have larger problems.
- Two interpretations: P1 and P2 view low reliability as an obstacle to be overcome (reliability is a necessary but not sufficient condition for validity); P5 views disagreement as a manifestation of the inherent pluralism of safety, and believes an ideal benchmark should "exhaustively cover" different perspectives rather than force consensus (Section 4.2).
Threats to External Validity and Lack of Maintenance
- What was measured: how developers understand threats to benchmark external validity (distribution drift, concept drift, data leakage, Goodhart's law, saturation) and whether they maintain benchmarks accordingly.
- Results: P1 and P2 described instances of distribution drift (such as the emergence of new types of harmful content and new methods for online communities to evade moderation); P1, P3, and P5 believe concept drift is an expected phenomenon in the safety domain. Although the interviews focused on concept drift, participants more often spontaneously mentioned data leakage (P1 said that once benchmark prompts are public they are "immediately captured by models"), Goodhart's law (P5), and saturation (P1, P3, P5).
- Authors' interpretation: these concerns did not prompt participants to actively maintain benchmarks—only P1 regularly updates; P3 lost motivation to maintain due to lack of adoption feedback, and P2 said it is difficult to obtain the new data needed to support updates. The authors believe that the lack of adoption and feedback directly weakens the incentives and resources for maintenance (Section 4.3).
Other Ablations and Analyses
- The paper has no ablation or parameter analysis experiments; Appendix A provides the full interview protocol (including specific questions for two hypothetical scenarios).
What are some points that can be further explored?
The authors acknowledge a small sample, selection bias, and self-report bias; the study is a qualitative interview study of 5 people, with no quantitative evaluation.
Limitations and Future Directions Noted by the Authors
- Small sample size: the authors state that due to time constraints and the difficulty of recruiting industry practitioners, there were only 5 participants, and the latter is a documented difficulty in research in this field (Section 5.3).
- Selection bias: the sample came mainly from the authors' professional networks and respondents recruited via social media, which the authors state introduced selection bias (Section 5.3).
- Self-report bias: as with other work relying on self-report, participants may be biased by a desire to appear competent (Section 5.3).
- Future directions: the authors state that future work could address these biases and strengthen the conclusions by recruiting more participants (including through academic-industry collaboration) and incorporating more data collection methods (Section 5.3).
- Call to the field: the authors call on industry to increase transparency about safety evaluation methods (such as publishing high-level descriptions of internal evaluation methods), to establish feedback channels on benchmark adoption through third-party organizations (such as NIST), and to establish academic-industry collaboration to develop shared standards for benchmark updating and retirement (Section 5.2).
Scope of Experimental Coverage
- The form of evidence is a qualitative analysis of 5 semi-structured interviews, with participants being 4 industry and 1 academic benchmark developers (Table 1).
- The participants' benchmark domains cover mental health, bias, toxicity, human well-being, pluralism, and evaluation methods (Table 1).
- The interview protocol explicitly limits the maintenance discussion to concept drift and does not cover contamination caused by training data leakage (Section 3); the paper does not report which specific benchmarks the participants developed or which models were involved.
- The paper does not report reliability metrics such as inter-coder agreement; coding disagreements were resolved through regular meetings of the research team (Section 3).
- The paper does not include quantitative experiments on any benchmark, model, or evaluation tool.
Summarize the main content of the paper
Interviews with 5 safety benchmark developers show that the intended use and actual use of benchmarks are disconnected, and that developers neither use nor maintain their own benchmarks.
This is a qualitative interview study on AI safety benchmark development practices, focusing on how developers understand the purpose and maintenance of benchmarks.
- Question: Safety benchmarks are described as a primary tool for understanding generative AI risks, but existing evidence shows they often fail to meet the evaluation needs of real deployment scenarios; the perspective of developers as a group has previously lacked research.
- Method: From November 2025 to January 2026, approximately 1-hour semi-structured interviews were conducted with 5 safety benchmark developers in academia and industry, coded using reflexive thematic analysis; the interviews focused on benchmark maintenance under concept drift, and hypothetical scenarios were used with participants who had not experienced drift.
- Finding one (purpose disconnect): The purposes developers envision are measuring progress, comparing models, and accountability, but they generally lack feedback on how benchmarks are actually used; industry participants themselves do not use public benchmarks when evaluating deployed systems, but instead rely on an internal evaluation ecosystem consisting of red teaming, user logs, and user feedback (Section 4.1).
- Finding two (reliability challenge): Inter-rater reliability is widely seen as a persistent challenge in safety benchmark development; P1 and P2 view it as an obstacle that must be overcome, while P5 views it as a signal of the inherent pluralism of safety and argues that benchmarks should cover multiple perspectives (Section 4.2).
- Finding three (maintenance gap): Developers acknowledge external validity threats such as distribution drift, concept drift, data leakage, Goodhart's law, and saturation, but only P1 regularly updates benchmarks; the others have never updated them since release; the authors argue that the lack of adoption feedback weakens the incentives and resources for maintenance (Section 4.3).
- Authors' conclusion: The authors argue that safety benchmarks should not only be grounded in deployment scenarios but also actively integrated into them, propose a bricolage-style evaluation model of bottom-up iterative refinement, and call on industry to increase transparency of evaluation methods, establish feedback channels through third-party organizations, and develop shared standards for benchmark updating and retirement (Section 5).