# LongMem: displaceable longterm artificial memory

Source: https://arxiv.org/abs/2306.07174
Kiwi News: https://news.kiwistand.com/stories/LongMem-displaceable-longterm-artificial-memory?index=0x648a3b58ee8033d5fd93cd638565bcb643ff0fae81a0ee9d948ffab2dbb61341ff6a776d
Submitted by m-j-r.eth on 2023-06-14 UTC, 1 upvote, 0 comments

## Summary

LongMem is a framework that proposes augmenting language models with long-term memory so they can use long past context despite fixed input-length limits. The authors (Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao and Furu Wei) design a decoupled architecture in which the original backbone LLM stays frozen as a memory encoder, while an adaptive residual side-network retrieves and reads from the memory. The framework can enlarge long-form memory to 65k tokens, and the article reports that it outperforms strong long-context models on the ChapterBreak benchmark and improves memory-augmented in-context learning. The code is open-sourced.

## Article

Augmenting Language Models with Long-Term Memory

Authors: Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, Furu Wei

Existing large language models (LLMs) can only afford fix-sized inputs due to the input length limit, preventing them from utilizing rich long-context information from past inputs. To address this, we propose a framework, Language Models Augmented with Long-Term Memory (LongMem), which enables LLMs to memorize long history. We design a novel decoupled network architecture with the original backbone LLM frozen as a memory encoder and an adaptive residual side-network as a memory retriever and reader. Such a decoupled memory design can easily cache and update long-term past contexts for memory retrieval without suffering from memory staleness. Enhanced with memory-augmented adaptation training, LongMem can thus memorize long past context and use long-term memory for language modeling. The proposed memory retrieval module can handle unlimited-length context in its memory bank to benefit various downstream tasks. Typically, LongMem can enlarge the long-form memory to 65k tokens and thus cache many-shot extra demonstration examples as long-form memory for in-context learning. Experiments show that our method outperforms strong long-context models on ChapterBreak, a challenging long-context modeling benchmark, and achieves remarkable improvements on memory-augmented in-context learning over LLMs. The results demonstrate that the proposed method is effective in helping language models to memorize and utilize long-form contents. Our code is open-sourced at this https URL.