One-year Summary of Real-world Feedback Cases on Performance of an Automated Seizure Burden Monitoring Algorithm
Abstract number :
3.253
Submission category :
3. Neurophysiology / 3B. ICU EEG
Year :
2024
Submission ID :
499
Source :
www.aesnet.org
Presentation date :
12/9/2024 12:00:00 AM
Published date :
Authors :
Presenting Author: Khalid Alsherbini, MD – University of Arizona, Banner health care
Tanaya Puranik, MS – Ceribell, Inc.
Baharan Kamousi, PhD – Ceribell, Inc.
Parshaw Dorriz, MD – Providence Mission Hospital Mission Viejo
Rationale: Clarity (Ceribell, Inc.) is an automated seizure-burden monitoring algorithm that runs on point-of-care electroencephalography (POC-EEG). In 2023, the company established a customer feedback system called Clarity Feedback. One goal of this program was to establish a pipeline to characterize disagreements between the on-site clinicians and Clarity, directly from commercial users. About a year after this feedback mechanism was added, we aim to summarize the type of cases that were submitted, their adjudication by a reviewer drawn from a panel of expert reviewers, and changes in these results using the latest version of the algorithm that was developed in the interim.
Methods: Cases submitted for Clarity Feedback were documented and sent post-hoc to an expert for online review. The reviewer was drawn from a panel of fellowship-trained epileptologists or clinical neurophysiologists. The reviewer provided their impressions, assigning a label to the recording based on ACNS guidelines: Electrographic Status Epilepticus (ESE), Seizure (Sz), or Not Status or Seizure. Additional information of other abnormal patterns was usually provided, which were broadly categorized as highly epileptiform patterns (HEP), abnormal or rhythmic activity. We summarized the cases reported and recorded from April 1st, 2023 to April 1st, 2024. We excluded files that were not reviewed by an epileptologist due to insufficient information. We ran post-hoc the latest version of the algorithm in the included files.
Results: A total of 233 EEGs were submitted for Clarity Feedback, of these 25 were used for algorithm training but are included in the dataset for summary and characterization. These cases represented 0.46% of the total of POC-EEG recordings collected during the analysis period. From the initial field reports,19 cases were submitted as possible false negatives (FNs) for ESE with a 90% seizure burden (SzB) threshold; 75 cases were reported as possible FNs for Sz with a 0% SzB threshold, and 81 were reported as possible false positives (FPs) for ESE with a 90% SzB threshold.
After expert review of all 233 EEGs, 7 of the cases were adjudicated as FN for ESE and 14 cases were adjudicated as FN for Sz. There were 71 FP cases after review. Using the latest version of the algorithm, we found trends of improvement in overcalls and undercalls by Clarity. The FN cases for ESE and Sz dropped by half (N = 3 and 6, respectively). The number of cases determined as FP for ESE decreased by more than half (N = 29 vs. 71).
Conclusions: Over the past year of Clarity feedback by customers, after more than 50,000 EEG recordings, only a small number of cases (233) were reported as disagreements. After careful review by experts in the field, only very few cases were considered true false negatives. Moreover, we found improvements with the latest version of the algorithm: we observed that the number of FNs of ESE and Sz, and FPs for ESE decreased to around half of the cases misclassified by the previous algorithm version.
Funding: Funded by Ceribell, Inc.
Neurophysiology