YaRN local LLMs (performant long context)

YaRN is a compute-efficient method for extending the context window of transformer language models that use Rotary Position Embeddings, which otherwise fail to generalize beyond their training sequence length. The authors, Bowen Peng, Jeffrey Quesnelle, Honglu Fan and Enrico Shippole, report that it requires 10x fewer tokens and 2.5x fewer training steps than previous methods. Using YaRN, they show that LLaMA models can use and extrapolate to context lengths much longer than their original pre-training allowed, surpassing previous state-of-the-art context extension methods. They also state that YaRN can extrapolate beyond the limited context of a fine-tuning dataset.