Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Zican Dong, Yupeng Hou, Beichen Zhang, Yingqian Min et al.
A comprehensive survey covering the entire process from pre-training to evaluation of LLMs, systematically organizing key techniques and future challenges.
The rapid advancement of LLMs has made it difficult for researchers to grasp the overall picture. Existing surveys focus on specific aspects or quickly become outdated, necessitating an integrated and up-to-date framework covering the full lifecycle of LLMs.
The authors set four key dimensions (pre-training, post-training, utilization, evaluation) and collect, classify, and analyze the latest research for each. Pre-training covers scaling laws, architectural innovations, and data curation; post-training covers SFT and RLHF; utilization covers ICL, prompt engineering, and agentic reasoning; evaluation covers language ability, reasoning, and safety benchmarks.
Provides a comprehensive framework that systematically surveys the current state and limitations of LLM research. In particular, it clearly identifies major challenges such as lack of theoretical understanding, efficiency issues in scaling, difficulties in alignment, and immaturity of agentic capabilities, thereby suggesting future research directions.