TL;DR
EvoComp is a framework that compresses visual tokens in Multimodal Large Language Models by retaining only semantically distinct key tokens, thereby improving inference speed while maintaining accuracy.
Problem
Multimodal Large Language Models (MLLMs) suffer from low inference efficiency in high-resolution or multi-image scenarios due to a large number of visual tokens. Existing compression methods based on attention or similarity heuristics often lead to accuracy degradation or insufficient compression rates.
Approach
Evolutionary Labeling Strategy: An evolutionary algorithm is designed to search for optimal token subsets that minimize the MLLM's output loss, enforcing semantic diversity through vocabulary-based token grouping.
Lightweight Compressor: An encoder-based transformer compressor is implemented to select the most informative and non-redundant visual tokens by jointly considering visual and textual contexts.
Tailored Loss Function: The compressor is trained using a combination of GHM loss to mitigate class and difficulty imbalance, and cosine similarity regularization to encourage semantic separation between retained and discarded tokens.
Results & Contribution
Outperforms existing heuristic-based methods across multiple vision-language benchmarks.
Retains 99.3% of the original accuracy under 3x token compression.
Achieves up to 1.6x inference speedup on mobile devices, demonstrating practical efficiency.