Can openness be too open? On the safety and trustworthiness of fully transparent LLMs

The paper argues that fully transparent large language models are not necessarily safer or more trustworthy. The authors review the relevant literature and point out that, because the reproducibility of LLM training pipelines is limited, users in practice cannot verify whether open weights correspond to the accompanying open data and training pipeline, and trusting the model still requires trusting the developers to act honestly, so the benefits of transparency are therefore limited. The paper also finds that even if a model is fully transparent and trustworthy, openness itself may benefit attackers, and open weights in particular raise attack success rates, with open data and pipelines having a similar effect. Although the same mechanisms can be used to improve models and defend against attacks, there is currently a lack of reliable lightweight mechanisms to patch LLM vulnerabilities, which limits the present benefits of openness. The authors note that the analysis is based on current LLM capabilities and analysis tools, and future developments may require reassessing these conclusions.