Jonathan F. Carter, Lionel Tarassenko
We propose Hypnos, a model that uses next-token prediction as a pretraining objective for multi-modal sleep physiological signals, learning superior representations compared to existing masked reconstruction or contrastive learning.
Physiological signals (EEG, ECG, respiration, etc.) are noisy and stochastic, and their semantic invariances are poorly understood, making existing masked reconstruction or contrastive learning unsuitable. Masked reconstruction is vulnerable to the stochastic nature of signals, while contrastive learning struggles with defining positive pairs.
Time-series data from eight sensor modalities (EEG, ECG, respiration, etc.) are tokenized into discrete token streams via residual vector quantization (RVQ). A large autoregressive RQ-Transformer is then trained to jointly predict the next token across all modalities in parallel. After pretraining, embeddings can be extracted from continuous sensor data of any subset of modalities for downstream tasks.
Hypnos significantly outperforms existing foundation models in sleep stage classification, matching strong supervised baselines with 100x less labeled data. It also generalizes to daytime physiology, surpassing a dedicated ECG foundation model in atrial fibrillation detection. This demonstrates that next-token prediction is a strong self-supervised objective for representation learning from multi-modal physiological signals.