Deployment-Aware Compression of Large Language Models for Edge Intelligence: A System-Level Survey

TL;DR

This survey systematically categorizes and analyzes compression techniques and system-level considerations for deploying large language models on edge devices.

Problem

When deploying large language models on edge hardware, there is a need to clearly understand and systematically organize the interaction between model compression algorithms and actual hardware/compiler toolchains. Existing benchmarks also have limitations in adequately evaluating robustness, numerical fidelity, and compiler-induced latency critical for real deployment.

Approach

The study classifies major compression strategies—quantization, pruning, and distillation—from a deployment-centric perspective and analyzes their interaction with various compilers like Tensor Virtual Machine, Core Machine Learning, and Tensor Runtime-LLM. Case studies on real deployment stacks illustrate how system-level constraints translate into concrete bottlenecks.

Results & Contribution

It presents a deployment-centric taxonomy and clearly identifies limitations of existing benchmarks, such as the lack of robustness-aware evaluation and explicit treatment of compiler-induced variability. The survey identifies open challenges related to continual adaptation, privacy-preserving personalization, and secure deployment, providing a structured synthesis for practitioners, compiler engineers, and hardware vendors.