TL;DR
A new framework is proposed to comprehensively evaluate LLMs' entire clinical consultation process using real EHR data across multiple dimensions.
Problem
Existing medical LLM evaluations are often static, single-turn, or outcome-focused, failing to adequately reflect the sequential, uncertain, and interactive nature of real-world clinical consultations. This risks overestimating or misjudging a model's actual clinical utility.
Approach
EHR-Grounded Evaluation Framework: Real patient EHR data is transformed into patient-specific knowledge graphs to build a simulation environment.
Multi-Turn Interaction Simulation: Models are evaluated on their ability to engage in multi-turn physician-patient dialogues.
8-Dimensional Competency Assessment: Models are comprehensively assessed across eight dimensions, including medical questioning skills, ethical conduct, explanation clarity, information integration, medication safety, handling ambiguous responses, information coverage, and diagnostic accuracy/reasoning.
Results & Contribution
Evaluation Results: LLMs performed well in medical questioning, ethical conduct, and explanation clarity but showed persistent weaknesses in handling ambiguous responses, information coverage, and diagnostic accuracy/reasoning.
Key Findings: Richer conversational context improved diagnostic reasoning but yielded limited gains in treatment planning. Recurrent interaction failures were observed, such as repetitive questioning, omission of past medical history, and inadequate handling of uncertainty.