Human oversight cannot defuse the risk of AI in war
Human oversight won’t solve AI warfare’s risks
Transformer published an article arguing that keeping humans in the decision loop does not eliminate the risks of AI involvement in military decisions, and that the real danger lies in human operators who over-rely on AI. The article cites a CNN report from September 18 that, during a war against Iran, the U.S. military prepared to board a Chinese cargo ship in the Middle East suspected of carrying nuclear components, based on a completely erroneous intelligence assessment produced by an analyst with the help of AI; aircraft had already taken off, and some said the incident nearly triggered war between two nuclear powers. Jacquelyn Schneider of Stanford University's Hoover Institution said AI makes analysts more confident in their assessments while seeing less of the data sources. The article also mentions Israel's Lavender system, which had an error rate of up to 10%, with personnel often rubber-stamping its decisions; Palantir's Maven Smart System was accused of being involved in the first-day bombing of an elementary school in Minab, killing more than 150 people, possibly due to outdated data, after which Palantir upgraded the system to require re-review of underlying intelligence.