Abstracts

Rationale: Identification of IEDs is challenging due to large patient diversity, expert shortage, and reader subjectivity. AI algorithms hold promise for overcoming these challenges. However, robust AI algorithms require a large, diverse set of labeled IEDs. In this study, we demonstrate a method that allowed 3 experts to annotate >10,000 patients’ EEGs in under 4 hours.

Methods: We utilized 11,212 scalp EEGs performed at MGH (age: 47 y median, range [2 d, 102 y], settings: 8,351 routine, 692 EMU, 2,169 NICU), and 93 EEGs (age >18 years, 30-60 min each) from Guinea. We first applied an automated IED detector SpikeNet to eliminate non-IEDs with prediction probabilities < 0.40. A two-dimensional embedding map using PaCMAP (Figure 1) for visualization and label spreading was computed using 58 IED features extracted from 825,770 events.
_x000D_ A MATLAB graphical user interface was developed to facilitate rapid IED annotation (Figure 1). Active learning was applied to iteratively annotate the dataset. In the first iteration we identified 15 cluster centers from IED and non-IED classes to be annotated. We asked each EEG expert to label these 30 events as IED vs non-IED. Next, we updated labels of the entire dataset such that any unlabeled event inherited a label from the nearest labeled sample. From iteration #2 onwards, in addition to labeling the next event furthest from any previously labeled events, we utilized an in-class clustering step to identify events where SpikeNet predictions disagree with the current labels assigned. After the expert reviewed those events (~60 per iteration), any unlabeled event inherited its label from the nearest labelled event.
_x000D_ To evaluate this method we conducted an experiment with three EEG experts (two neurologists and one experienced EEG researcher). Experts independently performed 40 iterations of labeling. Both time consumption and inter-rater reliability (IRR) measurements including percent agreement PA and Kappa κ (by Gwet’s AC1) were reported.

Results: As shown in Figure 2, the PA among three EEG experts was 82% on average, with range [80%, 84%]; and κ was on average 70%, with range [69%, 74%]. The IED IRR is “Substantial”, adopting standard naming conventions for levels of reliability indicated by κ values (Slight 0%-20%, Fair 21%-40%, Moderate 41%-60%, Substantial 61%-80%, and Almost-Perfect 81%-100%). EEG experts spent 3.6 hours (average) to label the entire dataset. The total time cost and number of events annotated by each EEG expert were: 1,938 events in 7.1 hours. After label spreading from labeled to unlabeled data using the PacMAP, the effective number of new labeled candidate IEDs is 825,770 from 11,305 patients.

Conclusions: Our results show that IEDs from >10,000 EEGs can be rapidly annotated by experts by organizing the labeling task in feature space rather than doing annotation in a standard serial fashion. Other uses for our method could be to help create large, diverse, comprehensive, and expert-validated database, that would ideally benefit the development of automated AI solutions. 

Funding: National Institutes of Health