Predicting the trajectory of intracranial pressure in patients with traumatic brain injury: evaluation of a foundation model for time series
Authors:
Florian D. van Leeuwen,
Shubhayu Bhattacharyay,
Alex Carriero,
Ethan Jacob Moyer,
Richard Moberg
Abstract:
Patients with traumatic brain injury (TBI) often experience pathological increases in intracranial pressure (ICP), leading to intracranial hypertension (tIH), a common and serious complication. Early warning of an impending rise in ICP could potentially improve patient outcomes by enabling preemptive clinical intervention. However, the limited availability of patient data poses a challenge in deve…
▽ More
Patients with traumatic brain injury (TBI) often experience pathological increases in intracranial pressure (ICP), leading to intracranial hypertension (tIH), a common and serious complication. Early warning of an impending rise in ICP could potentially improve patient outcomes by enabling preemptive clinical intervention. However, the limited availability of patient data poses a challenge in developing reliable prediction models. In this study, we aim to determine whether foundation models, which leverage transfer learning, may offer a promising solution.
△ Less
Submitted 18 October, 2024;
originally announced October 2024.
Empirical Evidence That There Is No Such Thing As A Validated Prediction Model
Authors:
Florian D. van Leeuwen,
Ewout W. Steyerberg,
David van Klaveren,
Ben Wessler,
David M. Kent,
Erik W. van Zwet
Abstract:
Background: External validations are essential to assess clinical prediction models (CPMs) before deployment. Apart from model misspecification, differences in patient population and other factors influence a model's AUC (c-statistic). We aimed to quantify variation in AUCs across external validation studies and adjust expectations of a model's performance in a new setting.
Methods: The Tufts-PA…
▽ More
Background: External validations are essential to assess clinical prediction models (CPMs) before deployment. Apart from model misspecification, differences in patient population and other factors influence a model's AUC (c-statistic). We aimed to quantify variation in AUCs across external validation studies and adjust expectations of a model's performance in a new setting.
Methods: The Tufts-PACE CPM Registry contains CPMs for cardiovascular disease prognosis. We analyzed the AUCs of 469 CPMs with a total of 1,603 external validations. For each CPM, we performed a random effects meta-analysis to estimate the between-study standard deviation $τ$ among the AUCs. Since the majority of these meta-analyses has only a handful of validations, this leads to very poor estimates of $τ$. So, we estimated a log normal distribution of $τ$ across all CPMs and used this as an empirical prior. We compared this empirical Bayesian approach with frequentist meta-analyses using cross-validation.
Results: The 469 CPMs had a median of 2 external validations (IQR: [1-3]). The estimated distribution of $τ$ had a mean of 0.055 and a standard deviation of 0.015. If $τ$ = 0.05, the 95% prediction interval for the AUC in a new setting is at least +/- 0.1, regardless of the number of validations. Frequentist methods underestimate the uncertainty about the AUC in a new setting. Accounting for $τ$ in a Bayesian approach achieved near nominal coverage.
Conclusion: Due to large heterogeneity among the validated AUC values of a CPM, there is great irreducible uncertainty in predicting the AUC in a new setting. This uncertainty is underestimated by existing methods. The proposed empirical Bayes approach addresses this problem which merits wide application in judging the validity of prediction models.
△ Less
Submitted 12 June, 2024;
originally announced June 2024.
