dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching

TL;DR

A training-free adaptive caching method that exploits the stability of prompts and responses to accelerate inference in diffusion LLMs.

Problem

Diffusion-based LLMs (dLLMs) suffer from high inference latency due to iterative denoising. Traditional KV-caching for ARMs is incompatible with dLLMs because of their bidirectional attention.

Approach

Observing that during dLLM inference, the prompt is static and most response tokens remain stable across adjacent denoising steps, we design dLLM-Cache, a training-free adaptive caching framework that combines long-interval prompt caching with feature similarity-based partial response updates.

Results & Contribution

On LLaDA 8B and Dream 7B models, dLLM-Cache achieves up to 9.1x FLOPs reduction on the LongBench-HotpotQA benchmark while maintaining output quality. It brings dLLM inference latency close to that of ARMs.