M. Mabrok
We model the hidden state space of LLMs as a Riemannian manifold and present a mathematical theory quantifying semantic distortion caused by token discretization, experimentally confirming universal geometric patterns across diverse models.
LLMs perform internal computations in continuous vector spaces but produce discrete tokens. The key issue is how this continuous-discrete mismatch affects the geometric structure of the representation space and how to quantify the resulting information loss.
The authors interpret LLM hidden states as points on a latent semantic manifold, a Riemannian submanifold equipped with the Fisher information metric. Each token corresponds to a Voronoi region partitioning the manifold. They define the expressibility gap to measure semantic distortion from vocabulary discretization and mathematically prove a rate-distortion lower bound on distortion for any finite vocabulary and a linear volume scaling law for the expressibility gap.
The theoretical predictions are validated across six transformer architectures (124M-1.5B parameters). Intrinsic dimensions exhibit an hourglass profile (high in encoder and decoder, low in middle layers), curvature structure is smooth, and the expressibility gap scales linearly with slopes 0.87-1.12 (R²>0.985). A persistent hard core of boundary-proximal representations invariant to model scale provides a geometric decomposition of perplexity. This work has implications for architecture design, model compression, decoding strategies, and scaling laws.