# Deepmind has a method for stealing LLM internals

Source: https://arxiv.org/abs/2403.06634
Kiwi News: https://news.kiwistand.com/stories/Deepmind-has-a-method-for-stealing-LLM-internals?index=0x660069c5899122e0e5ea871afe4f95456fc920e4b9ab4a049cdf70256e79f69774fb0047
Submitted by thatalexpalmer.eth on 2024-03-24 UTC, 1 upvote, 0 comments

## Summary

Researchers including Nicholas Carlini, Daniel Paleka and Florian Tramèr report a model-stealing attack that extracts precise, nontrivial information from black-box production language models such as OpenAI's ChatGPT and Google's PaLM-2. The attack recovers the embedding projection layer, up to symmetries, of a transformer model using typical API access. For under $20 USD, it extracts the entire projection matrix of OpenAI's Ada and Babbage models, confirming hidden dimensions of 1024 and 2048, respectively. The authors also recover the exact hidden dimension of gpt-3.5-turbo and estimate that recovering its entire projection matrix would cost under $2,000 in queries.

## Article

Stealing Part of a Production Language Model

Authors: Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Itay Yona, Eric Wallace, David Rolnick, Florian Tramèr

We introduce the first model-stealing attack that extracts precise, nontrivial information from black-box production language models like OpenAI's ChatGPT or Google's PaLM-2. Specifically, our attack recovers the embedding projection layer (up to symmetries) of a transformer model, given typical API access. For under \$20 USD, our attack extracts the entire projection matrix of OpenAI's Ada and Babbage language models. We thereby confirm, for the first time, that these black-box models have a hidden dimension of 1024 and 2048, respectively. We also recover the exact hidden dimension size of the gpt-3.5-turbo model, and estimate it would cost under $2,000 in queries to recover the entire projection matrix. We conclude with potential defenses and mitigations, and discuss the implications of possible future work that could extend our attack.