Adversarial examples in the ChatGPT era: why chatbot attacks differ from traditional adversarial attacks

SPY Lab writes that two basic principles of traditional adversarial example research (imperceptible perturbations and reliance on gradient optimization) largely do not hold for attacks against chatbots. Jailbreak attacks are initiated by users themselves, and third-party text in prompt injection is usually not seen by users, so attack text does not need to be imperceptible; the strongest current jailbreak methods are social-engineering-style instructions crafted by manual trial and error, not optimization algorithms. In a recent paper, the authors tested two optimization attacks, ARCA and GDBA, against GPT-2, LLaMa, and Vicuna fine-tuned for refusal, showing that optimization attacks are partially effective against undefended models but mostly fail against Vicuna, and even when prompts that trigger target outputs exist, they cannot be found.