AI Algorithms from RSNA challenge show potential in breast cancer detection on mammograms

Algorithms submitted to the Radiological Society of North America’s (RSNA) 2023 AI Challenge have demonstrated high performance in detecting breast cancer from mammography images, according to a new study published in Radiology, the RSNA’s official journal.

The RSNA Screening Mammography Breast Cancer Detection AI Challenge, held in 2023, saw participation from more than 1,500 teams worldwide. The challenge aimed to crowdsource artificial intelligence (AI) solutions that enhance automated cancer detection in screening mammograms. The goal was to support radiologists in improving workflow efficiency, patient care, and cost management.

Professor Yan Chen, a professor in cancer screening at the University of Nottingham, United Kingdom, led the analysis of the submitted algorithms. “We were overwhelmed by the volume of contestants and the number of AI algorithms that were submitted as part of the Challenge,” said Prof. Chen. “It’s one of the most participated-in RSNA AI Challenges. We were also impressed by the performance of the algorithms given the relatively short window allowed for algorithm development and the requirement to source training data from open-sourced locations.”

The challenge provided participants access to a dataset of approximately 11,000 breast screening images supplied by Emory University in Atlanta, Georgia, and BreastScreen Victoria in Australia. Participants were also permitted to use publicly available training data. A total of 1,537 functional algorithms were submitted and tested using a distinct set of 10,830 single-breast exams confirmed through pathology results.

On average, the algorithms achieved a specificity rate of 98.7 per cent for confirming the absence of cancer and a sensitivity rate of 27.6 per cent for identifying cancerous cases. The average recall rate was 1.7 per cent. When researchers combined the top three and top ten algorithms, sensitivity rose to 60.7 per cent and 67.8 per cent, respectively.

“When ensembling the top performing entries, we were surprised that different AI algorithms were so complementary, identifying different cancers,” said Prof. Chen. “The algorithms had thresholds that were optimised for positive predictive value and high specificity, so different cancer features on different images were triggering high scores differently for different algorithms.”

According to the research, the combined performance of the top ten algorithms is comparable to that of an average screening radiologist in Europe or Australia. Performance of individual algorithms varied depending on cancer type, imaging equipment manufacturer, and acquisition site. The algorithms were more effective at detecting invasive cancers than noninvasive ones.

Prof. Chen noted that since many of the AI models from the competition are open source, their availability could support the enhancement of future experimental and commercial tools for mammographic screening. “By releasing the algorithms and a comprehensive imaging dataset to the public, participants provide valuable resources that can drive further research and enable the benchmarking that is required for the effective and safe integration of AI into clinical practice,” she said.

The team intends to conduct further research to compare the challenge’s top algorithms against commercially available tools using a more extensive and diverse dataset. “Additionally, we will investigate the effectiveness of smaller, more challenging test sets with robust human reader benchmarks—such as those developed by the PERFORMS scheme, a UK-based program for assessing and assuring the quality of radiologist performance as an approach for AI evaluation, and compare its utility to that of large-scale datasets,” added Prof. Chen.