SliceGPT: Compress Large Language Models by Deleting Rows and Columns

TL;DR

SliceGPT is a post-training compression technique that reduces model size by deleting rows and columns of low importance from weight matrices in transformer models.

Problem

Large language models (LLMs) require enormous compute and memory resources. Existing sparsification techniques need additional data structures and offer limited speedup on current hardware. There is a need for methods that achieve practical inference speedup and memory savings while maintaining model performance.

Approach

SliceGPT leverages computational invariance in transformer networks. By exploiting properties of LayerNorm and residual streams, it transforms weight matrices so that deleting rows and columns preserves the output. Each weight matrix is replaced with a smaller dense matrix, reducing the embedding dimension and shrinking the entire model. This yields immediate speedup on standard hardware without requiring special sparsity support.

Results & Contribution

SliceGPT removes 25% of parameters from LLAMA2-70B, OPT 66B, and Phi-2 while maintaining 99%, 99%, and 90% of zero-shot performance, respectively. On 24GB consumer GPUs, it reduces total compute for LLAMA2-70B inference to 64% of the dense model; on 40GB A100 GPUs, to 66%. This is the first work to achieve significant reductions in actual GPU memory usage and inference time without additional code optimization.