TRADE: Transducer-Augmented Decoder for Speech LLM

TL;DR

TRADE augments a multimodal LLM with a transducer branch to combine frame-synchronous alignment with LLM linguistic reasoning, enabling both offline and streaming speech recognition with a single model.

Problem

Existing Speech LLMs use label-synchronous generation without acoustic-frame alignment, making real-time decoding and end-of-utterance detection difficult. Additionally, KV-cache memory grows linearly with utterance length for long-form speech.

Approach

TRADE addresses these issues with three design choices: (1) Tightly coupled dual vocabularies — a compact transducer vocabulary derived from the LLM vocabulary for zero-cost score fusion; (2) Chunk-synchronized streaming training with gradient stopping to eliminate train-inference mismatch; (3) Localized Decoder Audio Attention (LDAA), a causal sliding window that caps KV-cache memory independently of utterance length.

Results & Contribution

A single TRADE checkpoint supports offline (6.71% average WER on Open ASR Leaderboard) and streaming (8.40% WER with 960ms chunks) recognition. On long-form speech, it achieves 3.64% WER on TED-LIUM and 10.88% on Earnings-22 without external segmentation. It provides sentence-end punctuation timestamps that improve end-of-utterance detection F1 by +0.03 over acoustic VAD alone.