Multi-modal large language models for ophthalmology triage

Abstract

The increasing prevalence of ocular diseases worldwide and the resultant burden on healthcare systems has underscored the need for accurate, consistent, and scalable triage systems. This study explores the application of large vision-language models (VLMs) in improving diagnostic accuracy and, by extension, ophthalmic triage accu- racy. We propose a comprehensive framework that integrates structured clinical text generation from unstructured notes, supported by hallucination detection to ensure input reliability. To robustly evaluate diagnostic performance, we introduce a graph- based method that leverages a Directed Acyclic Graph (DAG) of medical concepts to compute dissimilarity scores between predicted and ground-truth diagnoses. This enables a more nuanced assessment of model output beyond exact label matching. We conduct a multimodal evaluation using various ophthalmic imaging modalities, com- paring text-only and image-assisted diagnoses. Our findings show that while image inputs can significantly enhance diagnostic accuracy for certain conditions, they may degrade performance in others—highlighting the need for context-aware integration of visual data. This work establishes a foundation for more interpretable and clinically aligned triage support systems powered by multimodal large language models (LLMs).