CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models

TL;DR

CosyVoice 2 is a scalable streaming speech synthesis model based on large language models, supporting both streaming and non-streaming synthesis in a single model, achieving human-level naturalness and minimal response latency.

Problem

Existing speech synthesis models required separate implementations for streaming (real-time) and non-streaming (high-quality) synthesis, with a trade-off between response latency and synthesis quality. Additionally, codebook utilization of speech tokens was low, reducing efficiency.

Approach

1) Introduced finite-scalar quantization to improve codebook utilization of speech tokens. 2) Streamlined the text-speech language model architecture to directly use a pre-trained LLM as the backbone. 3) Developed a chunk-aware causal flow matching model to support both streaming and non-streaming synthesis within a single model.

Results & Contribution

Trained on a large-scale multilingual dataset, CosyVoice 2 achieves human-level naturalness, minimal response latency, and virtually lossless synthesis quality in streaming mode. This is expected to significantly advance real-time conversational speech interfaces.