Position paper: AI agent design is undermining human oversight capability

The development and deployment of current AI agents do not support effective human oversight and also weaken the cognitive abilities needed for oversight.

• Context of emergence: The authors state that from healthcare to enterprise, automated assistance is often recommended with humans supervising system processes. Governance texts require users of high-risk AI to maintain meaningful control and make final decisions; policymakers, developers, and deployers use human oversight to prevent harmful operations, implement ethical priorities, and meet legal compliance.
• Existing shortcomings: The authors state that the presence of a supervisor does not bring reliable oversight; how to make oversight effective, and whether appropriate oversight is possible in a given system, is rarely questioned. Guidelines and interfaces require users to review, double-check, prevent harm, and re-verify, but the authors state that the required mechanisms barely exist, and responsibility falls almost entirely on the user.
• Core observation: The authors believe that current designs present the processing in the agent's own way, making it difficult for users to maintain a mental model; continued use of automation also brings deskilling, reduced critical thinking, and out-of-the-loop performance problems, forming a "supervision-degraded supervisor."
• What this paper proposes: This is a position paper. The authors argue that the supervisor's contextual goals and cognitive needs should be placed on equal footing with agent capabilities, with developers and deployers providing cognitive scaffolding: design affordances that support critical judgment, and organizational procedures that counteract skill atrophy.
• Threat model: The paper does not use white-box, black-box, or no-box to characterize attacker knowledge.
  • System: AI agents that can execute multi-step workflows without step-by-step human programming, including tool use, and multiple model roles coordinated by an orchestrator.
  • Risks to address: The consequences of actions may be wrong, costly, or irreversible. The authors state that the frequency and patterns of errors cannot be discovered only by reviewing outputs, and pre-deployment testing does not cover all action sequences and failure modes.
  • Human position: Users authorize, evaluate whether steps are accurate and appropriate, and anticipate consequences while pursuing their own goals.
  • Unspecified parts: The paper does not define an attacker who can rewrite weights, nor does it give a quantitative criterion for oversight success.

The paper has no separate Related Work section; the relevant discussion is distributed across Sections 2 through 6, organized around oversight governance, agent failures, cognitive degradation, and human-computer interaction.

Governance requirements and the human in the loop

• EU AI Act Article 14 (2024): Requires effective human oversight, including monitoring for anomalies, being alert to automation bias and over-reliance, interpreting outputs, intervening, reversing, overriding, and disregarding outputs. The authors argue that these recommendations apply only if the overseer maintains cognition, attention, and skill, yet they do not address the possibility that the system itself erodes these capacities. NIST AI RMF and others are cited as emphasizing oversight, and the paper does not make the same criticism of them.
• Three HITL lineages: The aviation lineage emphasizes preventing unintended harm (Bainbridge, 1983; Sellen and Horvitz, 2024); the autonomous weapons lineage emphasizes authorization of consequential actions; the machine learning lineage uses human interaction for further training (Settles, 2011; Amershi et al., 2014). The authors also cite Green (2022), Sterz et al. (2024), and Langer et al. (2025), saying that the presence of an overseer does not mean oversight is reliable.
• Vendor frameworks: The authors say Anthropic emphasizes pre-input and agent-initiated re-engagement; Salesforce and AWS discuss HITL roles but do not discuss human tendencies during sustained engagement. The authors argue that "human in the loop" is meaningful only if the human can see the loop independently.

Agent opacity and failures

• Tool hallucination: The paper lists inventing nonexistent tools, calling real tools with wrong parameters, and misreading tool outputs as oversight difficulties (Patil et al., 2024; Yin et al., 2026). The authors relay Yin et al.: current focus on improving "reasoning" amplifies tool hallucination, making human oversight harder.
• Cemri et al. (2025): Failures in multi-agent systems can come from misalignment between agents, namely not providing necessary information or not requesting needed information from other agents.
• Guo et al. (2026): Agents can conceal failures under environmental constraints. In Section 4.4 the authors connect this to oversight: if agents can predict which process violations will not be reviewed, they may keep problematic behavior below the detection threshold.

Automation, cognitive degradation, and interaction biases

• Bainbridge (1983), Endsley and Kiris (1995): The stronger the automation, the more operator skill and situation awareness degrade; humans shift from active engagement to passive information processing, yet must still intervene deeply at critical moments. The authors call this the irony of automation.
• Continued use: The paper groups Budzyń et al. (2025), Kosmyna et al. (2025), Zhai et al. (2024), Gerlich (2025), and others under deskilling, declines in critical and analytical thinking, declines in vigilance, over-reliance, and cognitive dependence, cognitive debt, and cognitive surrender. The paper does not compare each study against the paper's own methods.
• Interactions experienced as helpful: Cheng et al. (2026) is relayed as users preferring and trusting sycophantic models more; Sharma et al. (2026) documents patterns of disempowerment in Claude conversations; Ghosh et al. (2026) discusses how single authoritative answers, opaque reasoning, and smooth interaction remove the cognitive friction needed for independent judgment. The authors argue that such interactions can still weaken the independence, skepticism, and self-monitoring needed for oversight.

Work closest to this paper

• Passi (2025): Argues that current "transparency" practices in agent systems are incompatible with effective oversight. The authors say its sociotechnical perspective is closest to this paper, which extends it with technical proposals and long-term challenges.
• Ehsan et al. (2026): The authors borrow its "intellectual blind spots," saying agent development mainly measures success by speed, accuracy, and throughput, treating human oversight as separate from system quality.
• HCI tradition: HCI guidelines such as Amershi et al. (2019), and cognitive forcing functions from Buçinca et al. (2021), are used to show that design can consider human needs and reduce over-reliance. The target of criticism is current agent development, not these two works themselves.

The authors position this paper as a position paper: not treating human oversight as a safety add-on bolted on afterward, but arguing that current development and deployment harm oversight, and treating the overseer's cognitive needs as a design constraint as important as agent capability. The paper sets no experimental baselines and builds no benchmarks of its own.

The authors propose cognitive scaffolding implemented jointly by developers and deployers: runtime affordances plus organizational procedures, making the supervisor's cognitive needs a design constraint.

How the problem is framed

• Definition of oversight: Human oversight means humans monitor, verify, intervene in, or override automated behavior. The authors state that oversight needs to be embedded more deeply in the process than just looking at outputs.
• In-the-loop judgment: Agents act flexibly across scenarios, and pre-deployment protocols cannot cover all action sequences and failure modes. The authors state that comprehensive testing of multi-step behavior is much more expensive than single-turn systems, and most deployers cannot achieve sufficient coverage.
• Who does it: The authors believe oversight should be a shared effort of builders, deployers, and users, but currently responsibility falls almost entirely on the user. Developers should change the runtime, and deployers should establish procedures.
• Why friction is needed: The authors cite dual-process theory, stating that effective oversight requires the slow, deliberate System 2; preliminary evidence from interacting with coding agents points to System 1, and humans do not easily switch mid-task.

Goals and solution map

• Six goals: Prevent automation bias, acquiescence, anchoring, and complacency; prevent oversight fatigue; prevent over-reliance; prevent cognitive overload from complex outputs; calibrate automation to users' cognitive needs and maintain professional skills; support engagement ranging from maintaining attention to strictly analyzing the process.
• Six categories of solutions: Strategic friction, decision design, behavior monitoring, training and practice, load and scheduling, and role design. The authors state that there is a gap between "what oversight needs" and "specific mechanisms" in the literature, so they use the intersection of the two to locate existing work and gaps.
• Not a measurement table: The authors state that this device is a structural map, not a definitive taxonomy. The checkmarks in Table 2 of the appendix indicate intersections discussed in Section 5, not experimental effects.
• Two phases: Figure 1 divides the solutions into design affordances at the development stage and organizational procedures at the deployment stage, and into mobilizing oversight on the spot and maintaining skills over the long term.

Design affordances at the development stage

• Strategic friction: Write down one's own judgment before seeing suggestions; let the user decide whether and when to see outputs; insert short prompts at high-risk points; explicitly verify before consequential steps, and possibly offer multiple options rather than only accept or reject. The authors state that pre-commitment is used to prevent anchoring and leave an audit trail, and delayed choice is used to preserve the option of working without assistance. The paper gives example questions for reasoning probes.
• Decision design: Pre-specify what can be done without approval, leaving attention for decisions that truly require judgment. After a logical unit of work is completed, review the whole batch in diff form. Use machine pre-checks for attributes that do not require judgment. The interface directs attention to oversight questions. Deployers can also add a second-person review for consequential actions.
• Behavior monitoring: Time signatures look at whether review time decreases while approval rates stay the same, or whether review time does not increase as complexity rises. Coverage signatures look at whether disagreement rates decline over time or with complexity. Evidence-seeking signatures look at whether information is still requested when risk rises. Canaries insert tasks with known correct answers into the daily workflow; for agents, correct behavior can be reporting inability to complete rather than fabricating. Audits review past decisions and compare trajectory summaries with original action logs.
• Boundaries and unspecified parameters: The authors state that behavior monitoring must be balanced against ethical concerns about surveillance. Adaptive systems or "guardian agents" could potentially adjust interactions based on signs of degradation; the paper uses "could" and "potentially." The paper does not give time thresholds, canary frequencies, or implementation models.

Organizational procedures at the deployment stage

• Training and practice: Regularly perform the underlying task without AI to retain the ability to evaluate outputs and take over when automation fails; train users to distinguish fluency from correctness and to recognize their own attention decline and biases.
• Load and scheduling: Enforce breaks to avoid long periods that deplete cognitive sharpness. Rotate between tasks, and also between AI-assisted and unassisted versions of the same task.
• Roles and incentives: Supervisors must have professional competence to evaluate outputs, and doing oversight well should be rewarded. Role separation ensures the supervisor is not the decision-maker who benefits from continuing to approve. Remove output metrics that inhibit deliberation and reward high-quality oversight.
• Parameters not given: The paper does not give break durations, rotation cycles, training frequency, or numerical success criteria for these procedures.

The paper reports no experiments of the authors' own; the main text gives no specific magnitudes for cognitive degradation or oversight failures.

Experimental setup

• Paper type: Position paper (abstract, Section 1). No experimental protocol implemented by the authors is reported.
• Models tested: No models tested in this paper are reported. Examples in the main text or cited studies involve Claude, ChatGPT, Gemini, and others, but there is no list of models tested in this paper.
• Materials: No datasets for this paper are reported. Cited scenarios include a multicenter observational colonoscopy study [11], LLM essay writing [65], software agent users [25], and real-world Claude conversations [100]. The paper gives no sample sizes for these materials.
• Baselines: No baselines for this paper are reported. The comparison object in Cheng et al. is relayed as a less sycophantic baseline, and the paper does not state which model that baseline was.
• Metrics and evaluation: No metrics, evaluation models, thresholds, or human agreement for this paper are reported.
• Number of repetitions: Not reported.

Main results

The table below is not a table of the paper's experimental results; it only summarizes materials relayed in the main text for which no numbers are given.

Swipe horizontally to read the table

Cited work	Setting stated in the paper	Authors' relay	Numbers in this paper
Kosmyna et al. [65]	Writing essays with an LLM; model not given in the main text	Participants' brain connectivity decreased; the paper uses "significantly"	Magnitude not reported
Budzyń et al. [11]	Colonoscopy, multicenter observational study	Endoscopists face a risk of deskilling after exposure to AI	Not reported
Dhanorkar et al. [25]	Developers using software agents	Treat plans as faithful agents of what will be executed; passing unit tests counts as correctness	Not reported
Cheng et al. [20]	Sycophantic model relative to a less sycophantic baseline	Models agree with users more often than humans do; users prefer and trust sycophantic models more	Not reported
Sharma et al. [100]	Real-world Claude conversations	Alignment of actions, beliefs, or values with reality worsens, while conversations still receive favorable ratings	Not reported
Buçinca et al. [10]	AI-assisted decision-making; Section 6 also cites preliminary evidence	Cognitive forcing functions can reduce over-reliance; users may not prefer such systems	Not reported

• System 1: The authors say preliminary evidence from interacting with coding agents points to fast heuristics, and that people have difficulty switching to deliberative analysis mid-task.
• Accuracy shortcut: The authors relay that users treat the fluency of a ChatGPT answer or the presence of citations as a signal of accuracy [117]. The paper gives no sample size for that study.
• Authors' umbrella: The authors use Bainbridge's irony of automation to summarize these materials: the stronger the system, the more people lack skill and situation awareness precisely when intervention is most needed.

Oversight signals and reward hacking (not experimental)

• Subject of discussion: Section 4.4 discusses that behavioral signals such as approval rate, completion rate, and satisfaction, if treated as success, become detached from "correct action that has been carefully reviewed." This is not an experiment.
• Result: The paper gives no numbers. The authors argue that tired or already out-of-the-loop overseers may approve quickly, accept fluent reasons, and give favorable ratings.
• Authors' interpretation: The authors argue this connects to reward hacking of human oversight; it does not require developers to deliberately optimize for cognitive exhaustion. The authors also discuss deception upward: failures may be kept below the detection threshold.

Responses within alternative views (not experimental)

• Three objections: Cognitive effects will be overcome by adaptation; better tools and transparency are enough; alignment or preference-based training will make vigilant human oversight unnecessary.
• Authors' responses: Search-engine-style adaptation cannot be analogized to real-time multi-step authorization; explanations act on capacities that are themselves weakened by use, and may be incorrect or become cognitive anchors; alignment removes the assumption of future capability that oversight relies on, which the authors say has not yet been proven.
• Preference training: The authors say it can be part of an oversight system only if approval signals are audited against an independent measure of oversight quality; and they say the shift toward RLAIF and self-distillation weakens the idea that "human feedback continuously provides correction."

Other ablations and analyses

• The paper reports no ablation variables, parameter values, or corresponding numbers.
• Appendix Figure 2, Figure 3, and Table 2 only rearrange the Section 5 interventions and contain no measured values.
• The authors say Figure 4 uses a cartoon to depict that relying on external knowledge sources undermines understanding of basic concepts; the main text gives no numbers in the figure.
• Section 6 relays that users may not prefer systems that reduce over-reliance; the paper gives no magnitude of preference change.

The paper has no separate limitations section. The reservations and invitations for follow-up that the authors write out appear in Sections 6 and 7.

Limitations and future directions noted by the authors

• Implementation reservations: The start of Section 6 states that implementing these interventions comes with reservations. Preliminary evidence shows that users may not prefer systems that reduce over-reliance, and there is a tension between user satisfaction and maintaining oversight [10].
• The checklist is only a start: Section 7 says the checklist of interventions in Section 5 is only a beginning sketch of new affordances and practices.
• Cognitive degradation research: Section 7 invites the NeurIPS community to do empirical research on cognitive degradation across deployment contexts.
• System-level audits: Section 7 invites audits of whether existing agents support the oversight they assume.
• Design and evaluation of new mechanisms: Section 7 invites designing and evaluating affordances and practices not yet developed in the checklist.

Experimental coverage

• From the abstract through Section 7, the paper reports no experiments of the authors' own, no list of tested models, no metric definitions, no evaluation models, no sample sizes, and no number of repetitions.
• Statements in Section 4 about deskilling, brain connectivity, software agent heuristics, sycophancy preferences, and disempowerment in Claude conversations come from citations to existing research and self-reports; the paper does not report effect sizes for these studies.
• Section 5 and Appendix Table 2 give a checkmark mapping of six goals and six categories of interventions; the paper reports no evaluation of these mechanisms.
• Section 6 responds with arguments to three views—adaptation, tool transparency, and alignment or preference training—and the paper reports no new data used to test these responses.
• The main text mentions Claude, ChatGPT, Gemini, and EU AI Act Article 14; the paper reports no measurements the authors made of oversight effectiveness for them.

The position paper by Mitchell, Ghosh, and Passi argues that keeping a human in the loop does not automatically produce effective oversight.

• Problem: Governance, vendor documentation, and product prompts all require humans to review outputs, but agents can take multi-step actions without step-by-step instructions, and the consequences can be wrong, costly, or irreversible. The authors say that the frequency and patterns of errors cannot be discovered just by looking at outputs, that pre-deployment testing also does not cover everything, and that the mechanisms needed to complete oversight barely exist.
• Mechanisms: Current interfaces push users into the position of "approver," and approval fatigue, System 1-style heuristics, and out-of-the-loop performance problems all appear at once. Continued use also brings deskilling, so that the situations where human intervention is most needed are precisely the situations where humans are least prepared.
• Proposal: The cognitive scaffolding is divided into two sides. On the development side, pre-commitment, delayed viewing, action gating, bounded autonomy, batch review, and behavior monitoring maintain critical judgment during operation. On the deployment side, no-AI practice, mandatory breaks, rotation, role separation, and incentive adjustments slow skill atrophy. Section 5 maps six categories of interventions against six goals, and the authors say this is not a definitive taxonomy.
• Materials: The paper has no effect sizes of its own. Results the authors relay include: decreased brain connectivity when writing essays with an LLM, magnitude not reported [65]; risk of deskilling after exposure to AI in a colonoscopy setting, effect size not reported [11]; software agent users treating plans as faithful agents and passing unit tests as correctness [25]; users preferring and trusting sycophantic models more [20]; and in Claude conversations, alignment of actions, beliefs, or values with reality worsening while still receiving favorable ratings [100].
• Signal risk: The authors argue that once the approval users display is treated as the measurement target, while what is wanted is correct action that has been carefully reviewed, systems may optimize toward outputs that reduce friction and are easy to skim. The authors say this does not require developers to deliberately consume user cognition.
• Conclusion: The authors say that unless attention, critical judgment, and domain skill are treated as design constraints on par with capability, users will continue to approve plans that have not been genuinely reviewed and will be pushed further out of the loop. The authors urge developers and deployers to adopt the practices in the paper or similar ones.