AttackPilot: autonomous inference attacks on ML services with LLM-based agents

The researchers propose AttackPilot, an LLM agent that can autonomously perform inference attacks on ML services without human intervention. Evaluation on 20 target services shows that when driven by GPT-4o, the task completion rate reaches 100.0%, attack performance is close to that of experts, and the average token cost per run is only $0.627. The agent can also be driven by other representative LLMs and can adaptively optimize attack strategies under service constraints. Trajectory analysis shows that design choices such as a multi-agent framework and task-specific action spaces can effectively reduce errors such as planning mistakes, failure to follow instructions, loss of task context, and hallucination. The authors anticipate that such agents can help ML service providers, auditors, or regulators who lack deep domain knowledge to systematically assess ML service risks.