NEW YORK — What happened: A coalition of global radiology departments released an open benchmark testing seven leading multimodal vision-language models on chest X-rays, MRIs, and CT scans containing rare pathologies.
Why it matters: While AI models often ace pristine, curated benchmark datasets, their real-world reliability across different hospital imaging machines remains a major unanswered question in clinical practice.
Deep dive: When tested on standardized public datasets, the models achieved near-expert accuracy in detecting common conditions like pneumonia. However, when exposed to out-of-distribution scans from rural hospitals with older imaging equipment, sensitivity for rare interstitial lung diseases dropped by 34%.
Report check (claims vs what is verified vs still rumor): Commercial model releases claimed expert-level diagnostic parity across all global clinical settings. The benchmark verified this is only true when imaging hardware matches the specific training data distribution. Rumors that models could replace human radiologists for routine screenings have been dismissed by the American College of Radiology.
Open questions: What standardized calibration protocols are needed before imaging models can be safely deployed across diverse rural and urban healthcare networks?