Stealing part of a production-grade language model

AI summaryResearchers propose a model-stealing attack against production-grade language models that, using only regular API access, recovers the transformer's embedding projection layer up to symmetry, extracting the full projection matrices of OpenAI ada and babbage for less than $20. The authors claim this is the first attack that can exactly extract information from black-box production models such as ChatGPT and PaLM-2; it exploits the low-rank nature of the model's last layer projecting from the hidden dimension to a higher-dimensional logit space, recovering the hidden dimension and weight matrix through the logprobs and logit bias interfaces. On five OpenAI models, the extracted weights differ from the true weights by less than 7×10⁻⁴; the authors confirm that the hidden dimensions of ada and babbage are 1024 and 2048, respectively, and recover the hidden dimension size of gpt-3.5-turbo for less than $200, estimating that fully extracting its projection matrix would cost less than $2000.