How Confident Is Your AI? A MELD Project Study
Abstract number :
2.292
Submission category :
5. Neuro Imaging / 5A. Structural Imaging
Year :
2024
Submission ID :
855
Source :
www.aesnet.org
Presentation date :
12/8/2024 12:00:00 AM
Published date :
Authors :
Mathilde Ripart, MSc – UCL Great Ormond Street Institute of Child Health, London, UK
Hannah Spitzer, PhD – Institute for Stroke and Dementia Research, University Hospital, LMU Munich, Germany
Abdulah Fawaz, PhD – School of Biomedical Engineering & Imaging Sciences, King's College London, UK
Logan Z. J. Williams, PhD – School of Biomedical Engineering & Imaging Sciences, King's College London, UK
MELD Project, n.a. – UCL Great Ormond Street Institute of Child Health
Emma Robinson, PhD – School of Biomedical Engineering & Imaging Sciences, King's College London, UK
Juan Eugenio Iglesias, PhD – Massachusetts General Hospital & Harvard Medical School, USA
Konrad Wagstyl, MBPhD – School of Biomedical Engineering & Imaging Sciences, King's College London, UK
Presenting Author: Sophie Adler, MBPhD – UCL Great Ormond Street Institute of Child Health
Rationale: Focal cortical dysplasia (FCD) is a common cause of drug-resistant epilepsy, and accurate detection on MRI is critical for presurgical planning. However, identification of FCD remains challenging due to its subtle imaging features. AI algorithms have been proposed to automatically identify FCDs. However, due to the subtlety of the lesions and the lack of whole-brain context, they can identify areas of healthy cortex, false positives. This Multicentre Epilepsy Lesion Detection (MELD) project study aimed to 1) develop a whole-brain graph neural network (GNN) for segmenting FCDs and 2) output algorithm confidence scores.
Methods: Data from 20 centers were split equally at the center level into train and test cohorts, with data from three additional centers withheld for independent testing. A graph convolutional neural network (MELD Graph) was trained to identify FCDs on automatically extracted cortical features. Network performance was compared to a previously published, multi-layer perceptron algorithm. To compute confidence scores, for all predicted lesional vertices, we calculated the maximum model prediction score. The relative frequency of true positives among all predicted clusters was plotted against confidence scores to determine whether these were well calibrated using the Expected Calibration Error.
Results: On the independent test cohort (116 patients, 63 [53%] male, 22.5 [13.5-27.5] years) the Positive Predictive Value (PPV) was 0.77 (73% sensitivity, 56% specificity), compared to 0.5 (78% sensitivity, 59% specificity) using the baseline network. The significantly higher PPV was driven by a reduction in the number of false positive clusters. On MRI-negative patients, sensitivity was 65%. Per cluster confidence scores were well-calibrated, closely matching the relative frequency of true positives among predicted clusters in patients with an Expected Calibration Error of 0.10 (Figure 1). Examples of interpretable patient reports for high confidence and low confidence predictions are presented in Figure 2.
Conclusions: The MELD Project developed a state-of-the-art, interpretable and state-of-the-art graph neural network for FCD detection. The reduction in false positive clusters, alongside well-calibrated confidence scores and interpretable patient reports will facilitate clinical integration of lesion detection tools.
Funding: The MELD Project, M.R. and S.A. are funded by the Rosetrees Trust (A2665) and Epilepsy Research Institute (P2208). J.E.I is supported by the following grants: NIH 1RF1MH123195, 1R01AG070988, 1R01EB031114, 1UM1MH130981, 1RF1AG080371. K.W. is funded by the Wellcome Trust.
Neuro Imaging