Human-like sequential sound-to-meaning transfer drives artificial speech comprehension

TL;DR

Applying the human brain's local sequential phonology-to-semantics transformation mechanism to AI models improves speech comprehension performance.

Problem

AI speech comprehension models have reached human-level performance, but whether they rely on human-like mechanisms remains unknown.

Approach

Researchers used a phonology-semantics confusion paradigm comparing 12 speech language models with human brain data (SEEG). They identified two mechanisms in the human brain (local sequential transformation, global cross-regional hierarchy) and performed targeted lesioning and activation steering of model units.

Results & Contribution

Local sequential transformation predicted model performance, and manipulating those units improved performance. This mechanism was found across languages, suggesting a fundamental computational principle shared between biological and artificial intelligence.