Massive Activations in Large Language Models

TL;DR

Some activation values in LLMs are tens of thousands of times larger than others, acting as constant biases independent of input.

Problem

Understanding the internal workings of LLMs remains challenging. Specifically, the cause and role of neurons with extremely large values have not been clearly identified.

Approach

We analyzed activation values in various LLMs (Llama, Mistral, GPT, etc.) to identify neurons with extremely large values (massive activations), and investigated their locations and characteristics. We also experimented with whether these values change with input and how they affect the attention mechanism.

Results & Contribution

Massive activations exist in most LLMs, concentrated at specific positions in certain layers. These values are nearly constant regardless of input, acting as biases that concentrate attention probabilities on specific tokens. Similar phenomena were also observed in Vision Transformers. This finding provides important insights into understanding the internal mechanisms of LLMs.