Skip to content

Can AI judge you? An empirical audit of the deployment readiness of LLM-assisted hiring systems

Can AI Judge You? An Empirical Audit of Deployment Readiness in LLM-assisted Hiring Systems

arXiv: 其他 AI 安全论文· arXiv:2611.20140· Spandan Roy, Subhabrata Majumdar, Soudeep DebIndian Institute of Management BangaloreIndexed by this site · 2 days ago· Originally publishedDefenses· Its place in the scan record →

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

The authors audit the deployment readiness of an LLM résumé screening system: scalar scores are most sensitive to changes in qualifications, and RBAC plus request tagging can reduce the compliance rate of unauthorized internal requests from about 65% to 1%.

Questions
When delegating résumé screening to an LLM, organizations need to confirm that the system both distinguishes candidates based on evidence of qualifications and allows only authorized users to modify rules or approve exceptions; existing research mostly measures model bias or injection attacks, and lacks audit tools for the complete configured system.
Methods
The authors propose a reference architecture separating candidate evidence, evaluation, authority control, and evaluation records, and conduct a controlled audit along two dimensions: operational readiness (sensitivity of six decision interfaces to qualification changes, invariance to names and order) and compliance readiness (authority and provenance controls such as RBAC and Spotlighting, evaluation fidelity).
Experiments and results
210 resumes, 4 open-weight LLMs, and 16,800 evaluations show: scalar scores are most sensitive to qualification changes but poorly calibrated across models, and pairwise comparisons exhibit position bias; RBAC plus request marking reduces the compliance rate for unauthorized internal requests from about 65% to around 1%, the compliance rate for resume-embedded instructions varies widely across models; when decisions are reversed, the rationale almost always changes accordingly.
Limitations and future work
The authors note: the synthetic resumes cover a limited range of candidate experiences, occupations, and document formats; predictive validity for employment outcomes and actual recruiter use were not tested; compliance tests cover only predefined attacks and authority arrangements, excluding adaptive attackers, multi-turn interactions, and authorized-user misuse; operational and compliance requirements were not jointly evaluated under the same configuration.
Experimental setupMiniMax-M3、DeepSeek-V4-Pro et al. · 自构资历受控简历库(210 份简历、14 个职位描述、5 个行业) · 方向响应率(jobs responding %)、rubric 检出率 et al.
Threat model
Two types of unauthorized intervention: malicious internal users issuing GRANT/SHORTLIST/EXCLUSION requests beyond their authority (including impersonating administrators), and malicious candidates embedding steering instructions in the resume body (indirect prompt injection); the victim system is an LLM resume screener configured according to the reference architecture, with defenses at the application-layer governance layer (RBAC and Spotlighting request marking).
Evaluated models
MiniMax-M3DeepSeek-V4-ProNemotron-3-UltraGPT-OSS-120B
Benchmarks
自构资历受控简历库(210 份简历、14 个职位描述、5 个行业)
Metrics
方向响应率(jobs responding %)rubric 检出率较强简历选中率完整排序正确率balanced accuracy攻击成功率/不合规率评估不一致率
AI summaryThe researchers propose an LLM resume screening system architecture that separates candidate evidence, evaluation, permission control, and evaluation records, and empirically test its deployment readiness along the two dimensions of operation and compliance. In controlled experiments with 4 LLMs, 210 resumes, 14 job categories, and 6 decision interfaces, scalar scoring was generally better than simple pass/fail judgments at distinguishing changes in qualifications, while comparative judgments were sensitive to presentation order. Compliance tests show that explicit permission rules and separating resume content from instructions reduce compliance with unauthorized requests and with instructions embedded in resumes, but the effects vary across LLMs and use scenarios; the decisions of LLM evaluators are also often inconsistent with the reasons they give. The framework is used to audit fully configured resume evaluation systems to support deployment decisions.

The researchers propose an LLM resume screening system architecture that separates candidate evidence, evaluation, permission control, and evaluation records, and empirically test its deployment readiness along the two dimensions of operation and compliance. In controlled experiments with 4 LLMs, 210 resumes, 14 job categories, and 6 decision interfaces, scalar scoring was generally better than simple pass/fail judgments at distinguishing changes in qualifications, while comparative judgments were sensitive to presentation order. Compliance tests show that explicit permission rules and separating resume content from instructions reduce compliance with unauthorized requests and with instructions embedded in resumes, but the effects vary across LLMs and use scenarios; the decisions of LLM evaluators are also often inconsistent with the reasons they give. The framework is used to audit fully configured resume evaluation systems to support deployment decisions.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    How to configure and audit an LLM résumé screening system so that it faithfully interprets evidence while maintaining permission boundaries.

    Core question: How should organizations configure and evaluate LLM evaluators so that they faithfully interpret candidate evidence while maintaining governance over the authority of their judgments (the paper's research question).

    • Setting and importance: LLMs are moving from trials to routine organizational use, and increasingly deployments assign models evaluative roles (e.g., screening applicants against hiring requirements); the authors argue that delegating judgment means requiring both task competence and acting within the scope of authorization.
    • Shortcomings of existing approaches: The authors argue that existing governance research emphasizes configured sociotechnical arrangements but lacks tools to verify whether a specific LLM configuration acts according to the governance design; bias audits in hiring often report representational associations without establishing allocational consequences for decisions.
    • Core observation: An evaluator must distinguish "applying a decision rule" from "changing a decision rule"—the same instruction to "waive a hard qualification and shortlist" is an attack when issued by an ordinary evaluator but a legitimate exception when issued by an authenticated administrator.
    • Proposed approach: A reference architecture separating candidate evidence, evaluation, authority control, and evaluation records (Figure 1), and a diagnostic framework auditing deployment readiness along two dimensions: operational readiness (RQ1: the decision interface's responsiveness to qualification evidence and invariance to irrelevant cues) and compliance readiness (RQ2: authority and provenance controls, evaluation fidelity).
    • Threat model (Study 2):
      • Attacker goal: make the evaluator reach a decision favorable to the attacker (waive unmet qualifications, shortlist directly, or exclude qualified candidates).
      • Attacker identity and knowledge: malicious internal users (without the corresponding authority, able to impersonate an administrator) and malicious candidates (embedding instructions in the resume body).
      • Capabilities: internal users can issue GRANT/SHORTLIST/EXCLUSION requests; candidates can control resume text and embed steering instructions at the beginning, middle, or end.
      • Victim system: an LLM resume screener configured according to the reference architecture, with authentication information provided by the application layer outside the prompt.
  2. What related research exists?

    Spans four literature lines: organizational AI governance, LLM hiring bias, prompt injection security, and LLM reasoning fidelity.

    AI adoption and governance in organizations

    • Baird and Maruping (2021), Berente et al. (2021)—propose frameworks for delegating to agentic information systems and the interdependence of autonomy, learning, and inexplicability in AI management; this paper's architecture builds on the allocation of delegated responsibility.
    • Teodorescu et al. (2021), Wilson et al. (2025)—human-in-the-loop is not a uniform safeguard, and biased LLM recommendations can change reviewers' decisions; the authors accordingly argue that governance must attend to configured sociotechnical arrangements, but note that such work lacks tools to verify whether a specific configuration acts according to the governance design, which this paper's audit supplies.

    LLM use in hiring and bias audits

    • Armstrong et al. (2024), Yin et al. (2024), Wilson and Caliskan (2024), An et al. (2025)—name-substitution audits report racial and gender differences in GPT resume rankings; evidence from the Indian context is thin and inconsistent (Vijayaraghavan et al., 2025; Zaveri and Shah, 2025; Rao et al., 2025).
    • The authors identify two measurement problems: results depend on the elicitation instrument (prompt format sensitivity, position bias in comparative judgments; the preregistered audit of Vohra and Ravikiran (2026) finds ties and order preferences comparable to the demographic effects being measured); audits often report representational associations without establishing allocational consequences (Blodgett et al., 2020). Following the behavioral testing approach (Ribeiro et al., 2020), this paper tests the same qualification state across six decision interfaces and measures representational and allocational effects separately.

    LLM security vulnerabilities

    • Perez and Ribeiro (2022), Greshake et al. (2023)—direct and indirect prompt injection; Liu et al. (2024), Derczynski et al. (2024)—benchmarks and simulation tools for attacks and defenses.
    • Defense work: provenance marking by Hines et al. (2025) (delimiting, datamarking, encoding, i.e., Spotlighting), instruction hierarchy training by Wallace et al. (2024), external guardrails by Ayub and Majumdar (2024) and Vijil (2025). The authors note that these works frame instruction conflicts as adversary-versus-developer, whereas authority in organizational deployments is tiered, so this paper evaluates attack success rate together with authorized compliance.

    LLM reasoning fidelity

    • Jacovi and Goldberg (2020), Turpin et al. (2023)—LLM explanations may not reflect the factors driving outputs; Mökander et al. (2024), Raji et al. (2020)—AI audit frameworks. This paper audits the consistency between evaluations and rationales itself as a compliance property.

    Baselines and benchmarks: The protection conditions D-1/D0/D1/D2/D3 and C0–C4 in Study 2 are the paper's own control variants (Spotlighting taken from Hines et al., 2025); the resume bank construction method follows Castleman et al. (2026). Authors' positioning: this paper does not test model bias again, but translates organizational responsibility into testable requirements for the complete configured system.

  3. How does the paper address this problem?

    Use the reference architecture to separate evidence, authority, and records, then use controlled experiments to audit the decision interfaces and authority controls separately.

    Overall approach: treat deployment readiness as a property of the configured evaluation system (screening rules, prompts, decision interfaces, provenance handling, authority) rather than of a single LLM, and design two studies around the reference architecture in Figure 1: Study 1 tests the evaluation mechanism, Study 2 tests authority and provenance controls.

    Reference architecture (Section 3)

    • Five components: candidate intake and preparation, authority and provenance controls, LLM evaluation mechanism, evaluation records, downstream review and action; the organizational configuration specifies three elements: job requirements, screening rules, and authority.
    • Key design: user requests enter through a separate application interface; the application handles authentication and provides authority information, independent of any authority claimed in the request; candidate text serves only as evidence and has no authority to modify screening rules.
    • Evaluation records must distinguish "evidence-based evaluation" from "authorized exception," a distinction that motivates the fidelity audit in Study 2.

    Study 1: Resume bank and decision interfaces (Section 4)

    • Resume bank construction: 14 job descriptions (JDs) from public recruitment materials across 5 industries in India (IT, legal services, healthcare operations, medicine, education) (Table A1), structured with GPT-5.6 to determine required/preferred qualifications; GPT-5.6 then generates baseline resumes containing only the required qualifications.
    • Controlled variants: for each job, 3 required and 3 preferred qualifications serve as anchors, with 7 non-empty subsets each applied as deletion (required) or addition (preferred), yielding 15 states per job (including baseline), 210 resumes in total; deletions and additions are not mixed within the same variant (Appendix A.2).
    • Quality control: 15 domain experts compared baselines and variants using a slider; 79.6% of 186 judgments matched the expected direction (sign test p=0.00092), and mean JD applicability was 8.10/10 (Appendix A.3).
    • Six decision interfaces (Table 1): Binary (shortlist/reject), Ordinal (1–5 ordered categories), Scalar (0–100 score), Rubric (present/absent/unclear at anchors), Pairwise (two-resume comparison, tested in both orders), Listwise (four-resume ranking, all 24 permutations tested).
    • Quantitative analysis: linear regression on 0–100 scores and Firth logistic regression on shortlist decisions (Equation (1): b̂s = Bα̂(S) + Xβ̂(S) + Zθ̂(S), where B is 56 job-evaluator baseline indicators, X is 336 edit indicators, and Z is pairwise and three-way interaction terms); within each job-evaluator-intervention arm, test whether the three qualification effects are equal (H0: β1=β2=β3), with Holm correction across 112 joint tests.
    • Name experiment: SC/ST surnames and control surnames from 2011 Indian SECC data, paired with 40 male and 40 female names, 80 versions per resume tested on scalar scores; Appendix A.7 additionally reports a representational experiment (asking the LLM to generate names for occupations) and a parallel US surname study (400 names, Brunner-Munzel test, Holm correction).

    Study 2: Governance layer and attack conditions (Section 5)

    • Governance layer (Figure 7): takes input application context (hiring rules, AUTH_ID, authority mapping) and usage context (user request, resume text), matches authority, and constructs the final prompt; AUTH_ID is inserted by the application outside the user request, and the same identifier written by the user within the request is treated as ordinary unauthenticated text (Appendix B).
    • Five user-side conditions: D-1 (hiring rules only), D0 (no privileged access), D1 (D0+Spotlighting request marking), D2 (RBAC), D3 (RBAC+Spotlighting); Spotlighting replaces whitespace in user text with known delimiters.
    • Request types: GRANT (waive unmet requirements), SHORTLIST (shortlist regardless of qualifications), EXCLUSION (reject even if qualified); attackers include unauthorized internal users and administrator impersonators.
    • Five candidate-side conditions (Figure 9): C0 no protection, C1 warning, C2 warning+delimiters, C3 warning+datamarking, C4 warning+Base64 encoding; injected instructions are placed at the beginning, middle, or end of the resume, in three forms: direct outcome override, authority impersonation, and evidence-standard manipulation (examples in Appendix B.1).
    • Fidelity audit: compare decisions, qualification assessments (MET/NOT_MET), and rationales under ordinary versus adversarial requests, distinguishing "decision reversed" from "decision unchanged" outcomes.
  4. What experiments did the paper conduct?

    16,800 evaluations tested six interfaces; the governance layer reduced the unauthorized compliance rate from 65.1% to 1.1%, but when the decision was reversed, the rationale always changed.

    Experimental setup

    • Models tested: 4 open-weight LLMs—MiniMax-M3, DeepSeek-V4-Pro, Nemotron-3-Ultra, GPT-OSS-120B; GPT-5.6 was used only to structure JDs and generate baseline resumes.
    • Data: 210 resumes (14 JDs × 15 qualification states), 5 industries; 16,800 resume evaluations in Study 1, 20 calls per resume (temperature=0).
    • Study 2 sample: 56 resumes (4 per JD: 2 baselines and their weaker variants with 3 required qualifications deleted).
    • Metrics: directional response rate (proportion of the 14 JDs whose mean changed in the expected direction), rubric anchor detection rate, stronger-resume selection rate, full-ranking accuracy, balanced accuracy, attack success rate/non-compliance rate, rationale inconsistency rate.
    • Evaluation method: qualification ordering was validated by 15 domain experts (79.6% directional agreement, sign test p=0.00092); statistical tests included HC3 covariance, Firth penalized likelihood ratio, and Holm multiple correction (112 joint tests).
    • Baselines/controls: baseline resumes in Study 1; D-1 (no provenance control) and C0 (no protection) conditions in Study 2.

    Main results

    Study 2 user-side attack success rate (Figure 8A, SHORTLIST requests, aggregated across 4 LLMs):

    ConditionConfigurationAttack success rate
    D-1Hiring rules only65.1%
    D0No privileged access5.9%
    D1D0+request marking5.9%
    D2RBAC3.3%
    D3RBAC+request marking1.1%
    • The authors report that under D3 the evaluator recognizes valid AUTH_IDs, with compliance rates for authorized users far higher than for ordinary evaluators (Figure 8B: GRANT authorized 94.9%, EXCLUSION authorized 89.9%, unauthorized only 6.6% and 0.9%).
    • When impersonating an administrator (Figure 8C), the GRANT deception success rate under D2 was 15.4%, dropping to 0.9% under D3; EXCLUSION under D3 was 0.0%.
    • By model (Figure 8D), under D3, 3 of 4 LLMs reached a 100.0% compliance rate for authorized administrator requests, and GPT-OSS-120B was 0.0% for both unauthorized and deceptive requests.

    Decision interface sensitivity (Study 1, Figure 4, Table A2)

    • When deleting 3 required qualifications: the scalar (0–100) interface had the highest response rate across all 4 LLMs (92.9%–100.0%), followed by ordinal, with binary lowest (14.3%–85.7%, Nemotron-3-Ultra only 14.3%).
    • Rubric detection: detection rate for added preferred qualifications was 98.2%–100.0%; detection rate for deleted required qualifications ranged from 26.0% for MiniMax-M3 to 86.9% for DeepSeek-V4-Pro.
    • Regression results (Figure 6): among 56 job-evaluator combinations, required-qualification effects were not all equal in 39 combinations under scalar scores, versus only 15 under binary decisions; the authors accordingly argue that the binary interface is insufficient to reflect qualification changes.

    Order bias and calibration

    • Pairwise (Figure 5A): selection rate was higher when the stronger resume was placed second, with Nemotron-3-Ultra rising from 31.9% to 89.6% and DeepSeek-V4-Pro from 39.5% to 84.7%; these two models had selection rates below 50% when the stronger resume was placed first.
    • Listwise (Figure 5B): the strongest resume was most likely to be selected when placed last, with selection rate decreasing the earlier the position (e.g., as shown by the annotated values in the required-deletion arm for MiniMax-M3 ranging from 93.5% in first position to 98.9% in last position, full-ranking accuracy varied with position across models).
    • Calibration (Table 2): the scalar thresholds best matching binary decisions ranged from 53 to 73, corresponding to balanced accuracy of 78.4%–98.7%; the authors state that the same score means different things across LLMs and recommend small-scale pilot calibration for each specific job.

    Name associations and fidelity

    • Name experiment: the authors report no statistically significant association between surname category and shortlist status/score, but individual combinations differed widely (e.g., GPT-OSS-120B gave "Pooja Nadar" 55 and "Pooja Duraisamy" 78 after deleting two required qualifications, with both at 85 at baseline); in Appendix A.7, Nemotron-3-Ultra had 29 significant group-gender comparisons, versus 0 for the other three models.
    • Fidelity (Table 3, 2,191 pairwise comparisons where the initial decision was opposite to the attack goal): when the decision was reversed, 93/93 (100.0%) showed a new rationale problem; when the decision was unchanged, 8/2,098 (0.4%) showed a rationale problem. The authors state that resisting the request itself does not guarantee consistent evaluation, and a successful attack can create a contradiction between decision and rationale.

    Other ablations and analyses

    • Resume-embedded instructions (Figure 10): under C0, Nemotron-3-Ultra's non-compliance rate was 95.9%–100.0%; C3 (warning+datamarking) reduced MiniMax-M3 to 4.2%–8.3% and DeepSeek-V4-Pro to 4.8%–7.1%, but Nemotron-3-Ultra still had 48.0%–59.5% under C3; GPT-OSS-120B was 0.0% from C1 onward (C4 middle position 38.5%, 13/84).
    • Injection position: for most models, the non-compliance rate was highest when the instruction was at the beginning of the resume (Top column in each panel of Figure 10).
    • Representational experiment (Table A5): among LLM-generated names for orthopedic surgeons, women accounted for 0.00%–0.24%, and for obstetrics and gynecology 91.13%–99.97%; SC/ST surnames averaged 7.61%–11.30% (the SECC 2011 proportion is about 29.43%).
    • Response rates for each interface under single/double anchor deletion are shown in Table A2 (e.g., at k=1, binary response rate was 0.0%–71.4%).
  5. What are some points that can be further explored?

    The authors list limitations such as synthetic resume coverage and adaptive attackers, and propose directions for joint evaluation and field research.

    Limitations and future directions noted by the authors

    • Coverage of synthetic résumés (Section 6.3): Controlled synthetic résumés facilitate interpretable comparisons, but the candidate experiences, occupations, and document formats covered are limited.
    • Predictive validity and manner of use (Section 6.3): The experiments did not establish predictive validity for employment outcomes, nor did they examine how recruiters use the evaluation results.
    • Attack coverage (Section 6.3): The compliance tests cover established attacks and permission arrangements, not adaptive attackers, multi-turn interactions, or abuse by authorized users.
    • Joint evaluation (Section 6.3): The authors consider it urgent to jointly examine operational and compliance requirements under the same configuration—controls that reduce unauthorized influence may also affect the interpretation of legitimate evidence or the response to permitted requests.
    • Field studies (Section 6.3): Future work could examine whether explicit verification tasks and independent exception records improve reviewers' ability to detect errors and correct evaluations; broader coverage of languages, occupations, and intersecting demographic signals would help establish the scope of applicability of the conclusions.
    • Risk levels and allocation of responsibility (Section 6.3): The authors state that acceptable risk levels and optimal allocation of responsibility require further research.

    Scope of experimental coverage

    • The models tested were 4 open-weight LLMs (MiniMax-M3, DeepSeek-V4-Pro, Nemotron-3-Ultra, GPT-OSS-120B); GPT-5.6 was used to generate résumés and JDs; the paper does not report results for closed-source commercial models.
    • Study 1 covered 14 JDs, 5 Indian industries, 210 résumés, and 16,800 evaluations (20 calls per résumé, temperature=0); the name experiment covered Indian SC/ST surnames and control surnames, and Appendix A.7 additionally contains a parallel study with US surnames.
    • Study 2 covered 56 résumés, 5 user-side configurations (D-1 to D3), 5 résumé-side defenses (C0 to C4), 3 injection positions, and 3 types of injection instructions; the fidelity audit was based on 2,191 pairwise comparisons.
    • The paper explicitly excludes upstream résumé extraction and preprocessing, authentication implementation, recruiter behavior, and downstream hiring outcomes from its scope (Section 3.2).
    • The paper does not report the specific sample size for each attack condition (except that some cells for GPT-OSS-120B in Figure 10 are labeled with a denominator of 84) or the number of repeated calls in Study 2.
  6. Summarize the main content of the paper

    A framework for auditing the deployment readiness of LLM resume screening systems, covering decision interfaces, permission controls and evaluation fidelity.

    This is an empirical audit paper on the deployment readiness of LLM-assisted hiring systems, with authors from the Indian Institute of Management Bangalore.

    • Problem: When delegating resume screening to LLMs, organizations need evidence that the system both distinguishes candidates by qualification evidence and responds only to authorized interventions; the authors formalize these as operational readiness (RQ1) and compliance readiness (RQ2).
    • Method: Propose a reference architecture separating candidate evidence, evaluation mechanism, authority control, and evaluation records; conduct 16,800 evaluations with 4 open-weight LLMs on 210 qualification-controlled resumes (14 JDs, 5 industries, validated by 15 experts), testing six decision interfaces; then test unauthorized internal requests and resume-embedded instructions with a governance layer consisting of RBAC and Spotlighting request marking.
    • Interface results: When deleting 3 required qualifications, scalar scores had the highest response rate (92.9%–100.0%) and binary the lowest (14.3%–85.7%); however, scalar scores were poorly calibrated across models (thresholds 53–73); in pairwise comparisons, selection rate was higher when the stronger resume was placed second (e.g., Nemotron-3-Ultra rising from 31.9% to 89.6%).
    • Compliance results: Without provenance controls, the success rate of unauthorized SHORTLIST requests was 65.1%, dropping to 1.1% with RBAC plus request marking (D3), and 3 of 4 models reached a 100.0% compliance rate for authorized administrator requests; protection against resume-embedded instructions varied by model, with Nemotron-3-Ultra still showing a 48.0%–59.5% non-compliance rate under warning plus datamarking.
    • Fidelity results: Among 2,191 pairwise comparisons, 93/93 showed a new rationale problem when the decision was reversed, versus only 8/2,098 when the decision was unchanged; the authors state that resisting a request does not equal consistent evaluation.
    • Authors' conclusion: Deployment readiness must be evaluated for the specific configuration (model, prompt, interface, rules, authority); the authors recommend that hiring teams calibrate score thresholds for each specific job, implement RBAC and separately delimit user text and resume text, and, in settings where evaluation rationales matter, test and even train models to produce valid decisions and valid explanations.
Read originalarxiv.org