# YaRN local LLMs (performant long context)

Source: https://arxiv.org/abs/2309.00071
Kiwi News: https://news.kiwistand.com/stories/YaRN-local-LLMs-performant-long-context?index=0x65468abe3e2b307b6b916fa5072854ffab433bf182ac49fa39588c688dfb491ca89763d2
Submitted by m-j-r.eth on 2023-11-04 UTC, 1 upvote, 0 comments

## Summary

YaRN is a compute-efficient method for extending the context window of transformer language models that use Rotary Position Embeddings, which otherwise fail to generalize beyond their training sequence length. The authors, Bowen Peng, Jeffrey Quesnelle, Honglu Fan and Enrico Shippole, report that it requires 10x fewer tokens and 2.5x fewer training steps than previous methods. Using YaRN, they show that LLaMA models can use and extrapolate to context lengths much longer than their original pre-training allowed, surpassing previous state-of-the-art context extension methods. They also state that YaRN can extrapolate beyond the limited context of a fine-tuning dataset.

## Article

YaRN: Efficient Context Window Extension of Large Language Models

Authors: Bowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico Shippole

Rotary Position Embeddings (RoPE) have been shown to effectively encode positional information in transformer-based language models. However, these models fail to generalize past the sequence length they were trained on. We present YaRN (Yet another RoPE extensioN method), a compute-efficient method to extend the context window of such models, requiring 10x less tokens and 2.5x less training steps than previous methods. Using YaRN, we show that LLaMA models can effectively utilize and extrapolate to context lengths much longer than their original pre-training would allow, while also surpassing previous the state-of-the-art at context window extension. In addition, we demonstrate that YaRN exhibits the capability to extrapolate beyond the limited context of a fine-tuning dataset. Code is available at this https URL