AI safety cannot rely solely on constraining system prompts; it also requires evaluating the model's actual output

AI safety evaluation cannot look only at system prompt text; it must also examine the model's output behavior in actual deployment. Although system prompts can improve transparency, they provide only weak guarantees about model behavior; the randomness of LLMs means the same instruction has varying effects across tasks and contexts. xAI once revoked unauthorized prompt changes to Grok, and OpenAI also adjusted system prompts due to GPT-4o's sycophancy problem. The authors recommend verifying system behavior through risk evaluation, adversarial testing, and cross-language audits, and retesting after updates.