Deepmind has a method for stealing LLM internals

Researchers including Nicholas Carlini, Daniel Paleka and Florian Tramèr report a model-stealing attack that extracts precise, nontrivial information from black-box production language models such as OpenAI's ChatGPT and Google's PaLM-2. The attack recovers the embedding projection layer, up to symmetries, of a transformer model using typical API access. For under $20 USD, it extracts the entire projection matrix of OpenAI's Ada and Babbage models, confirming hidden dimensions of 1024 and 2048, respectively. The authors also recover the exact hidden dimension of gpt-3.5-turbo and estimate that recovering its entire projection matrix would cost under $2,000 in queries.