Abstracts

Rationale:
This study aims to investigate the use of e-diaries only to forecast seizures within the next 24-hour period, with multiple models. Recent work has shown that seizure frequency affects metrics. Additionally, it has been shown that the moving average is a better benchmark model to evaluate model performance¹.




Methods:
We evaluate the seizure forecasting performance of several models on a retrospective Seizure Tracker dataset. There are 5501 patients in the dataset (3300 patients in the training set, 1100 patients in the validation set, and 1101 patients in the testing set). The models we tested are cycle², generalized linear model (GLM), perceptron, multilayer-perceptron (MLP), and 1D convolutional neural network (Conv1D). The performance metrics we used are Brier score, area under curve of receiver-operating curve (AUCROC) and area under curve of precision-recall curve (AUCPR), summarized within individual seizure frequency bins. The statistical significance between models and moving average was tested by MANOVA for each metric.




Results:
The metrics of different models against monthly seizure frequency are shown in Figure 1 and the statistical test results are shown in Table 1. Though some models show significantly different metrics against moving average, there is no model with all three metrics showing significantly different simultaneously.

Figure 1. The metrics of interest of the selected model. The mean and 95% confidence interval are indicated by solid line and shaded area respectively. The marker size indicates the normalized number of diaries within each seizure frequency bin.






Conclusions:
This study tested various types of models, from simple to complex, on a large-scale real-world seizure diary dataset. The most complex models were flexible enough to learn extremely complex patterns from the data even if they were not specified ahead of time. Our results show that none of models were significantly different from the moving average across all metrics simultaneously. Thus, the simplest model (moving average) would be considered optimal. The absolute performance of a moving average is well below what would be considered clinically useful. Therefore, using e-diaries alone may be insufficient to forecast seizures within the next 24-hour period.









Reference




1. Chang, C., B. Zhang, R. Moss, R. Picard, M. B. Westover, and D. Goldenholz. Necessary for seizure forecasting outcome metrics: seizure frequency and benchmark model. medRxiv. 2024. doi:10.1101/2024.05.15.24307446.




2. Karoly PJ, Cook MJ, Maturana M, et al. Forecasting cycles of seizure likelihood. Epilepsia. 2020;61(4):776-786. doi:10.1111/epi.16485




Funding:
NIH NINDS K23NS124656