Abstracts

Rationale: Low power is a problem in many fields, as underpowered studies that find a statistically significant result will exaggerate the magnitude of the observed effect size. We quantified the statistical power and magnitude error of studies of epilepsy surgery outcomes.


Methods: We extracted publication level data from a Cochrane review of epilepsy surgery (West, et al., 2019). The “ground truth” for each publication was the pooled odds ratio from a meta-analysis for each clinical variable, computed with package meta in software R (version 4.2.1). We calculated the power of each study to detect the “ground truth” effect size, assuming a Chi-squared test, using R package exact. We computed the error from the “ground truth” odds ratio to the observed odds ratio and then compared this to the error of the odds ratio estimated using Bayesian logistic regression and a neutral prior distribution. That prior distribution was derived from an a large multi-center study by the European Brain Bank consortium (Lamberink, et al., 2020).


Results: The median power across all studies was 14%. Studies with a median sample size or less (n< =56) and a statistically significant result exaggerated the true effect size by a factor of 5.4 (median odds ratio 9.3 vs. median true odds ratio 1.7), while the Bayesian estimate of the odds ratio only exaggerated the true effect size by a factor of 1.6 (2.7 vs. 1.7).


Conclusions: The typical study of epilepsy surgery outcomes is statistically underpowered. We conclude that Bayesian estimation of odds ratio attenuates the exaggeration of significant effect sizes in underpowered studies. Bayesian estimation may improve patient counseling about the chance of seizure freedom after epilepsy surgery.


Funding: A.S.D. was supported by the National Center for Advancing Translational Sciences of the NIH under award number UL1 TR002378 and KL2 TR002381.