Learning to Forget: Sleep-Inspired Memory Consolidation for Resolving Proactive Interference in Large Language Models

TL;DR

A method inspired by sleep memory consolidation to resolve context interference in LLMs.

Problem

LLMs are vulnerable to proactive interference (PI): outdated information in the context window disrupts retrieval of current values. This occurs regardless of context length and resists prompt engineering.

Approach

SleepGate introduces three mechanisms into the KV cache: (1) a conflict-aware temporal tagger to identify stale entries, (2) a lightweight forgetting gate to selectively evict/compress stale entries, and (3) a consolidation module to merge surviving entries into compact summaries. These activate periodically during inference, governed by an adaptive entropy-based trigger. Training uses a dual-phase objective combining language modeling and post-sleep retrieval.

Results & Contribution

In experiments with a small-scale transformer, SleepGate achieves 99.5% retrieval accuracy at PI depth 5 and 97.0% at depth 10, while all baselines remain below 18%. Theoretically, it reduces the interference horizon from O(n) to O(log n).