[2307.02486] LongNet: Scaling Transformers to 1,000,000,000 Tokens

LongNet is a Transformer variant that scales sequence length to more than 1 billion tokens without sacrificing performance on shorter sequences, according to the paper by Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng and Furu Wei. It introduces dilated attention, which expands the attentive field exponentially as the distance grows. The paper states that LongNet has linear computation complexity and a logarithmic dependency between any two tokens, and that its dilated attention can replace standard attention in existing Transformer optimization. Experiments reported in the paper show strong performance on long-sequence modeling and general language tasks.