Evaluating large language model workflows in clinical decision support for triage and referral and diagnosis

Analysis

Why This Paper Matters

Accurate medical decision-making remains a critical challenge for both patients and clinicians. Patients often struggle to interpret symptoms and determine appropriate care pathways, while clinicians face the burden of integrating complex data for timely diagnoses. This paper addresses this gap by evaluating large language models (LLMs) and a retrieval-augmented generation (RAG) workflow on a curated dataset of 2000 medical cases from the MIMIC database. The work is timely given the rapid adoption of LLMs in healthcare and the need for rigorous benchmarking in clinical decision support.

The significance lies in its focus on real-world clinical tasks—triage, referral, and diagnosis—rather than narrow NLP benchmarks. By testing multiple LLM versions and a RAG-enhanced pipeline, the study provides a comparative view of how these models perform on tasks that directly impact patient outcomes. This positions the paper as a practical resource for AI practitioners deploying LLMs in clinical settings.

Technical Contributions

• Curated dataset: 2000 medical cases derived from MIMIC, designed to evaluate triage, referral, and diagnosis tasks.
• Multi-model benchmarking: Comparison of several LLM versions (exact models not specified in abstract) to assess performance variability.
• RAG integration: Incorporation of retrieval-augmented generation to ground LLM outputs in external medical knowledge, potentially improving accuracy and reducing hallucination.
• Task-specific evaluation: Assessment of models on three clinical decision tasks: likely diagnosis, specialist suggestion, and urgency assessment.

Results

The abstract reports that LLMs and the RAG workflow are capable of providing personalized insights into likely diagnoses, suggesting appropriate specialists, and assessing urgent care needs. The models also show potential to support clinicians in refining diagnoses and decision-making. However, no concrete metrics (e.g., accuracy, precision, recall, F1 scores) or comparisons against human clinicians or baselines are provided in the abstract. The lack of quantitative results limits the ability to assess the magnitude of improvement or clinical readiness.

Significance

This paper contributes to the growing body of evidence that LLMs can augment clinical decision-making, particularly in triage and referral contexts where patient-clinician communication gaps are common. The use of RAG is a notable technical choice, as it addresses a key weakness of LLMs—lack of domain-specific knowledge—by retrieving relevant medical information. For AI practitioners, the study underscores the importance of task-specific evaluation and the potential of hybrid LLM-RAG systems in healthcare. However, the absence of detailed metrics and human baselines means further validation is required before clinical deployment. The work sets a foundation for future studies to quantify the clinical impact of LLM-based decision support tools.