AI misjudged intelligence and nearly triggered a U.S.-China conflict

1. 23:04
   Transformer

   Human oversight cannot defuse the risk of AI in war

   Transformer published an article arguing that keeping humans in the decision loop does not eliminate the risks of AI involvement in military decisions, and that the real danger lies in human operators who over-rely on AI. The article cites a CNN report from September 18 that, during a war against Iran, the U.S. military prepared to board a Chinese cargo ship in the Middle East suspected of carrying nuclear components, based on a completely erroneous intelligence assessment produced by an analyst with the help of AI; aircraft had already taken off, and some said the incident nearly triggered war between two nuclear powers. Jacquelyn Schneider of Stanford University's Hoover Institution said AI makes analysts more confident in their assessments while seeing less of the data sources. The article also mentions Israel's Lavender system, which had an error rate of up to 10%, with personnel often rubber-stamping its decisions; Palantir's Maven Smart System was accused of being involved in the first-day bombing of an elementary school in Minab, killing more than 150 people, possibly due to outdated data, after which Palantir upgraded the system to require re-review of underlying intelligence.

1. 17:00
   The Guardian · 人工智能

   Timnit Gebru and Emily M. Bender: Don't just focus on "superintelligence" — error-prone AI nearly triggered a third world war this month

   Timnit Gebru and Emily M. Bender point out that CNN reported on September 18 that the U.S. military relied on intelligence generated by error-prone chatbots, mistakenly concluding a Chinese ship was carrying nuclear weapons components, and only discovered the error just before an imminent interdiction, nearly triggering a U.S.-China conflict and even a third world war. The two say such large language models are essentially "stochastic parrots," whose capabilities and risks were already well documented in the 2021 paper "On the Dangers of Stochastic Parrots," and that the real danger comes from people mistakenly believing they possess superintelligence. They call for stronger regulation of "AI" systems in high-risk scenarios such as military and medical settings, and for accountability to rest with people who hold decision-making power.

Current attention36·Peak in the comparable period36（Oct 7 12:00）·Change over the comparable past 24 hours 0%