VLMs for X-ray image analysis

Abstract

X-ray imaging is a widely used diagnostic tool in healthcare to identify various medical conditions. However, the manual interpretation of chest x-ray images is time-consuming and requires significant expertise. With increasing demand, this process can lead to delays in diagnosis, which may affect timely patient care. This project addresses these challenges by developing an end-to-end system that integrates both disease localisation and interpretive reporting for chest x-ray images. A Faster R-CNN model was fine-tuned on a subset of the VinDr-CXR dataset to detect and localise thoracic diseases, achieving a mean Average Precision (mAP@0.5) of 0.4180. The localised regions were then passed to CXR-LLaVA to generate clinical interpretations based on zero-shot prompting. This system is deployed using a web application for real-time visualisation and interpretive reporting of chest x-ray analysis. By bridging the gap between disease localisation and clinical explanation, this project aims to streamline and enhance the efficiency of diagnostic workflows in radiology.