D. Alber, Zihao Yang, A. Alyakin, E. Yang, Sumedha Rai, Aly A. Valliani, Jeff Zhang, Gabriel R. Rosenbaum et al.
Medical LLMs can be made to generate medical errors by poisoning only 0.001% of training data, and a biomedical knowledge graph-based detection method can block 91.9% of harmful content.
LLMs trained on massive web data are vulnerable to deliberate medical misinformation, especially in healthcare where false information can compromise patient safety. The actual risk of data-poisoning attacks and detection methods have not been sufficiently studied.
We injected medical misinformation into The Pile dataset at a rate of 0.001% of tokens and fine-tuned LLMs. We measured how often poisoned models generated errors on medical questions and compared their performance on standard benchmarks. We also proposed a harm mitigation algorithm using biomedical knowledge graphs (e.g., UMLS) to validate LLM outputs.
Poisoning just 0.001% of data caused models to produce significantly more medical errors, while performance on standard benchmarks remained nearly unchanged, making detection difficult. The proposed knowledge graph-based detection method captured 91.9% of harmful content with an F1 score of 85.7%. This study quantitatively demonstrates the risk of data poisoning in medical LLMs and provides a practical defense strategy.