ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models

TL;DR

This paper presents an extension of Schedule-Free Learning for large-scale LLM training, demonstrating significant performance gains over existing schedules.

Problem

While Schedule-Free Learning has shown promise, its strong performance for LLM training has only been demonstrated at small scales. Scaling it to larger batch sizes and model sizes requires specific fixes.

Approach

The authors identify necessary fixes for scaling and present ScheduleFree+, a learning-rate-free and schedule-free method. They also provide a theoretical foundation for model averaging and checkpoint merging during pretraining.

Results & Contribution

ScheduleFree+ greatly outperforms Warmup-Stable-Decay (WSD) schedules. It is most effective for long-duration training, outperforming state-of-the-art schedules by 31% at 1000 tokens per parameter. The work provides a theoretical basis for using model averaging and checkpoint merging.