Dyke Ferber, O. E. El Nahhas, G. Wölflein, I. Wiest, J. Clusmann, M. Leßmann, S. Foersch, Jacqueline Lammert et al.
An autonomous AI agent integrating GPT-4 with precision oncology tools (vision transformers, MedSAM, OncoKB, etc.) significantly improved clinical decision-making accuracy in oncology.
Clinical decision-making in oncology requires integrating multimodal data (histopathology, radiology, genomics) and multidomain expertise. Existing LLMs alone have low accuracy and are difficult to integrate into real clinical workflows.
We developed an autonomous AI agent based on GPT-4, designed to autonomously use vision transformers for detecting MSI and KRAS/BRAF mutations from histopathology slides, MedSAM for radiological image segmentation, and web search tools including OncoKB, PubMed, and Google. The agent was evaluated on 20 multimodal patient cases for tool use accuracy, clinical conclusion accuracy, and guideline citation accuracy.
The AI agent achieved 87.5% tool use accuracy, 91.0% correct clinical conclusions, and 75.5% accurate guideline citations. Compared to GPT-4 alone, decision-making accuracy improved from 30.3% to 87.2%, demonstrating that integrating LLMs with precision oncology tools substantially enhances clinical accuracy.