MIT Technology Review: We over-rely on AI's ability to refuse
We’re putting too much faith in AI’s ability to say no
MIT Technology Review published an article pointing out that refusal has become a load-bearing wall of AI safety, but this mechanism is probabilistic in nature and is destined to be difficult to make fully reliable. In 2021, Anthropic proposed that large language models should be useful, honest, and harmless. Today, models have learned to refuse a large number of prompts through red-team testing and AI-to-AI training, yet they still retain knowledge for wrongdoing, with capability and benevolent intelligence growing in tandem. The article states that some users are trying to use the most advanced AI to cultivate biological pathogens and build autonomous drone swarms, while the refusal boundary is secretly drawn by AI companies, and governments will also step in.