AI Hallucination Intelligence Nearly Triggered U.S. Military Boarding Action on Chinese Vessel

CNN reported that this spring, a U.S. military armed operation targeting a Chinese vessel was urgently called off after aircraft had already taken off, because the intelligence the operation was based on was fabricated by an AI chatbot. An analyst at the Special Operations Command used an AI chatbot to synthesize open-source data with classified signals intelligence, and the chatbot misjudged the vessel's cargo manifest; the analyst used the tool again to organize the erroneous conclusion into a seemingly formal summary, which was circulated through command channels. The intelligence report claimed the ship carried components for a nuclear weapons program, at a time when the United States was at war with Iran. Jake Steckler, a GovAI research scholar and former U.S. Army officer, told TechCrunch that military personnel need to understand the inherent uncertainty of LLMs, especially in decisions that could lead to the use of force, such as targeting, intelligence analysis, or operational planning. He believes this incident should prompt more AI guardrails rather than avoidance of AI, but if adoption speed is prioritized above all else, it could make military personnel lose trust in the system and actually slow adoption.