💡 The paper proposes a lightweight memory mechanism called delta-mem to enhance large language models by providing a compact online state of associative memory. The problem addressed is the need for large language models to accumulate and reuse historical information in long-term assistants and agent systems, which is challenging due to the high cost of expanding the context window and ineffective context utilization.

The proposed method, delta-mem, augments a frozen full-attention backbone with a compact online state that compresses past information into a fixed-size state matrix updated by delta-rule learning. This online state is used to generate low-rank corrections to the backbone's attention computation during generation, allowing for efficient online memory.

The results show that delta-mem improves the average score of the frozen backbone and achieves larger gains on memory-heavy benchmarks, such as MemoryAgentBench and LoCoMo, while preserving general capabilities. Notably, delta-mem achieves these results with only an 8x8 online memory state, demonstrating that effective memory can be realized through a compact online state directly coupled with attention computation, without requiring full fine-tuning, backbone replacement, or explicit context extension. Overall, the paper contributes a novel and efficient approach to enhancing large language models with online memory, which has the potential to improve performance in a range of applications.