-
A burn-in(g) question: How long should an initial equal randomization stage be before Bayesian response-adaptive randomization?
Authors:
Edwin Y. N. Tang,
Stef Baas,
Daniel Kaddaj,
Lukas Pin,
David S. Robertson,
Sofía S. Villar
Abstract:
Response-adaptive (RA) trials offer the potential to enhance participant benefit but also complicate valid statistical analysis and potentially lead to a higher proportion of participants receiving an inferior treatment. A common approach to mitigate these disadvantages is to introduce a fixed non-adaptive randomization stage at the start of the RA design, known as the burn-in period. Currently, i…
▽ More
Response-adaptive (RA) trials offer the potential to enhance participant benefit but also complicate valid statistical analysis and potentially lead to a higher proportion of participants receiving an inferior treatment. A common approach to mitigate these disadvantages is to introduce a fixed non-adaptive randomization stage at the start of the RA design, known as the burn-in period. Currently, investigations and guidance on the effect of the burn-in length are scarce. To this end, this paper provides an exact evaluation approach to investigate how the burn-in length impacts the statistical properties of two-arm binary RA designs. We show that (1) for commonly used calibration and asymptotic tests an increase in the burn-in length reduces type I error rate inflation but does not lead to strict type I error rate control, necessitating exact tests; (2) the burn-in length substantially influences the power and participant benefit, and these measures are often not maximized at the maximum or minimum possible burn-in length; (3) the conditional exact test conditioning on total successes provides the highest average and minimum power for both small and moderate burn-in lengths compared to other tests. Using our exact analysis method, we re-design the ARREST trial to improve its statistical properties.
△ Less
Submitted 25 March, 2025;
originally announced March 2025.
-
Exact statistical tests using integer programming: Leveraging an overlooked approach for maximizing power for differences between binomial proportions
Authors:
Stef Baas,
Yaron Racah,
Elad Berkman,
Sofía S. Villar
Abstract:
Traditional hypothesis testing methods for differences in binomial proportions can either be too liberal (Wald test) or overly conservative (Fisher's exact test), especially in small samples. Regulators favour conservative approaches for robust type I error control, though excessive conservatism may significantly reduce statistical power. We offer fundamental theoretical contributions that extend…
▽ More
Traditional hypothesis testing methods for differences in binomial proportions can either be too liberal (Wald test) or overly conservative (Fisher's exact test), especially in small samples. Regulators favour conservative approaches for robust type I error control, though excessive conservatism may significantly reduce statistical power. We offer fundamental theoretical contributions that extend an approach proposed in 1969, resulting in the derivation of a family of exact tests designed to maximize a specific type of power. We establish theoretical guarantees for controlling type I error despite the discretization of the null parameter space. This theoretical advancement is supported by a comprehensive series of experiments to empirically quantify the power advantages compared to traditional hypothesis tests. The approach determines the rejection region through a binary decision for each outcome dataset and uses integer programming to find an optimal decision boundary that maximizes power subject to type I error constraints. Our analysis provides new theoretical properties and insights into this approach's comparative advantages. When optimized for average power over all possible parameter configurations under the alternative, the method exhibits remarkable robustness, performing optimally or near-optimally across specific alternatives while maintaining exact type I error control. The method can be further customized for particular prior beliefs by using a weighted average. The findings highlight both the method's practical utility and how techniques from combinatorial optimization can enhance statistical methodology.
△ Less
Submitted 17 March, 2025;
originally announced March 2025.
-
Thompson, Ulam, or Gauss? Multi-criteria recommendations for posterior probability computation methods in Bayesian response-adaptive trials
Authors:
Daniel Kaddaj,
Lukas Pin,
Stef Baas,
Edwin Y. N. Tang,
David S. Robertson,
Sofía S. Villar
Abstract:
To implement a Bayesian response-adaptive trial it is necessary to evaluate a sequence of posterior probabilities. This sequence is often approximated by simulation due to the unavailability of closed-form formulae to compute it exactly. Approximating these probabilities by simulation can be computationally expensive and impact the accuracy or the range of scenarios that may be explored. An altern…
▽ More
To implement a Bayesian response-adaptive trial it is necessary to evaluate a sequence of posterior probabilities. This sequence is often approximated by simulation due to the unavailability of closed-form formulae to compute it exactly. Approximating these probabilities by simulation can be computationally expensive and impact the accuracy or the range of scenarios that may be explored. An alternative approximation method based on Gaussian distributions can be faster but its accuracy is not guaranteed. The literature lacks practical recommendations for selecting approximation methods and comparing their properties, particularly considering trade-offs between computational speed and accuracy. In this paper, we focus on the case where the trial has a binary endpoint with Beta priors. We first outline an efficient way to compute the posterior probabilities exactly for any number of treatment arms. Then, using exact probability computations, we show how to benchmark calculation methods based on considerations of computational speed, patient benefit, and inferential accuracy. This is done through a range of simulations in the two-armed case, as well as an analysis of the three-armed Established Status Epilepticus Treatment Trial. Finally, we provide practical guidance for which calculation method is most appropriate in different settings, and how to choose the number of simulations if the simulation-based approximation method is used.
△ Less
Submitted 29 November, 2024;
originally announced November 2024.
-
Data Augmentation in Graph Neural Networks: The Role of Generated Synthetic Graphs
Authors:
Sumeyye Bas,
Kiymet Kaya,
Resul Tugay,
Sule Gunduz Oguducu
Abstract:
Graphs are crucial for representing interrelated data and aiding predictive modeling by capturing complex relationships. Achieving high-quality graph representation is important for identifying linked patterns, leading to improvements in Graph Neural Networks (GNNs) to better capture data structures. However, challenges such as data scarcity, high collection costs, and ethical concerns limit progr…
▽ More
Graphs are crucial for representing interrelated data and aiding predictive modeling by capturing complex relationships. Achieving high-quality graph representation is important for identifying linked patterns, leading to improvements in Graph Neural Networks (GNNs) to better capture data structures. However, challenges such as data scarcity, high collection costs, and ethical concerns limit progress. As a result, generative models and data augmentation have become more and more popular. This study explores using generated graphs for data augmentation, comparing the performance of combining generated graphs with real graphs, and examining the effect of different quantities of generated graphs on graph classification tasks. The experiments show that balancing scalability and quality requires different generators based on graph size. Our results introduce a new approach to graph data augmentation, ensuring consistent labels and enhancing classification performance.
△ Less
Submitted 20 July, 2024;
originally announced July 2024.
-
Exact statistical analysis for response-adaptive clinical trials: A general and computationally tractable approach
Authors:
Stef Baas,
Peter Jacko,
Sofía S. Villar
Abstract:
Response-adaptive clinical trial designs allow targeting a given objective by skewing the allocation of participants to treatments based on observed outcomes. Response-adaptive designs face greater regulatory scrutiny due to potential type I error rate inflation, which limits their uptake in practice. Existing approaches for type I error control either only work for specific designs, have a risk o…
▽ More
Response-adaptive clinical trial designs allow targeting a given objective by skewing the allocation of participants to treatments based on observed outcomes. Response-adaptive designs face greater regulatory scrutiny due to potential type I error rate inflation, which limits their uptake in practice. Existing approaches for type I error control either only work for specific designs, have a risk of Monte Carlo/approximation error, are conservative, or computationally intractable. To this end, a general and computationally tractable approach is developed for exact analysis in two-arm response-adaptive designs with binary outcomes. This approach can construct exact tests for designs using either a randomized or deterministic response-adaptive procedure. The constructed conditional and unconditional exact tests generalize Fisher's and Barnard's exact tests, respectively. Furthermore, the approach allows for complexities such as delayed outcomes, early stopping, or allocation of participants in blocks. The efficient implementation of forward recursion allows for testing of two-arm trials with 1,000 participants on a standard computer. Through an illustrative computational study of trials using randomized dynamic programming it is shown that, contrary to what is known for equal allocation, the conditional exact Wald test based on total successes has, almost uniformly, higher power than the unconditional exact Wald test. Two real-world trials with the above-mentioned complexities are re-analyzed to demonstrate the value of the new approach in controlling type I errors and/or improving the statistical power.
△ Less
Submitted 17 March, 2025; v1 submitted 1 July, 2024;
originally announced July 2024.
-
X-CBA: Explainability Aided CatBoosted Anomal-E for Intrusion Detection System
Authors:
Kiymet Kaya,
Elif Ak,
Sumeyye Bas,
Berk Canberk,
Sule Gunduz Oguducu
Abstract:
The effectiveness of Intrusion Detection Systems (IDS) is critical in an era where cyber threats are becoming increasingly complex. Machine learning (ML) and deep learning (DL) models provide an efficient and accurate solution for identifying attacks and anomalies in computer networks. However, using ML and DL models in IDS has led to a trust deficit due to their non-transparent decision-making. T…
▽ More
The effectiveness of Intrusion Detection Systems (IDS) is critical in an era where cyber threats are becoming increasingly complex. Machine learning (ML) and deep learning (DL) models provide an efficient and accurate solution for identifying attacks and anomalies in computer networks. However, using ML and DL models in IDS has led to a trust deficit due to their non-transparent decision-making. This transparency gap in IDS research is significant, affecting confidence and accountability. To address, this paper introduces a novel Explainable IDS approach, called X-CBA, that leverages the structural advantages of Graph Neural Networks (GNNs) to effectively process network traffic data, while also adapting a new Explainable AI (XAI) methodology. Unlike most GNN-based IDS that depend on labeled network traffic and node features, thereby overlooking critical packet-level information, our approach leverages a broader range of traffic data through network flows, including edge attributes, to improve detection capabilities and adapt to novel threats. Through empirical testing, we establish that our approach not only achieves high accuracy with 99.47% in threat detection but also advances the field by providing clear, actionable explanations of its analytical outcomes. This research also aims to bridge the current gap and facilitate the broader integration of ML/DL technologies in cybersecurity defenses by offering a local and global explainability solution that is both precise and interpretable.
△ Less
Submitted 2 June, 2024; v1 submitted 1 February, 2024;
originally announced February 2024.
-
Constrained Markov decision processes for response-adaptive procedures in clinical trials with binary outcomes
Authors:
Stef Baas,
Aleida Braaksma,
Richard J. Boucherie
Abstract:
A constrained Markov decision process (CMDP) approach is developed for response-adaptive procedures in clinical trials with binary outcomes. The resulting CMDP class of Bayesian response -- adaptive procedures can be used to target a certain objective, e.g., patient benefit or power while using constraints to keep other operating characteristics under control. In the CMDP approach, the constraints…
▽ More
A constrained Markov decision process (CMDP) approach is developed for response-adaptive procedures in clinical trials with binary outcomes. The resulting CMDP class of Bayesian response -- adaptive procedures can be used to target a certain objective, e.g., patient benefit or power while using constraints to keep other operating characteristics under control. In the CMDP approach, the constraints can be formulated under different priors, which can induce a certain behaviour of the policy under a given statistical hypothesis, or given that the parameters lie in a specific part of the parameter space. A solution method is developed to find the optimal policy, as well as a more efficient method, based on backward recursion, which often yields a near-optimal solution with an available optimality gap. Three applications are considered, involving type I error and power constraints, constraints on the mean squared error, and a constraint on prior robustness. While the CMDP approach slightly outperforms the constrained randomized dynamic programming (CRDP) procedure known from literature when focussing on type I and II error and mean squared error, showing the general quality of CRDP, CMDP significantly outperforms CRDP when the focus is on type I and II error only.
△ Less
Submitted 28 January, 2024;
originally announced January 2024.
-
A stochastic programming approach for dynamic allocation of bed capacity and assignment of patients to collaborating hospitals during pandemic outbreaks
Authors:
Stef Baas,
Sander Dijkstra,
Richard J. Boucherie,
Anne Zander
Abstract:
Sustaining regular and infectious care during an infectious outbreak requires adequate management support for capacity allocation for regular and infectious patients. During the COVID-19 pandemic, hospitals faced severe challenges, including uncertainty concerning the number of infectious patients needing hospitalization and too little regional cooperation. This led to inefficient usage of healthc…
▽ More
Sustaining regular and infectious care during an infectious outbreak requires adequate management support for capacity allocation for regular and infectious patients. During the COVID-19 pandemic, hospitals faced severe challenges, including uncertainty concerning the number of infectious patients needing hospitalization and too little regional cooperation. This led to inefficient usage of healthcare capacity. To better prepare for future pandemics, we have developed a decision support system for central regional decision-making on opening and closing (regular care) hospital rooms for infectious patients and assigning new infectious patients to regional hospitals. Since the relabeling of rooms takes some lead time, we make decisions with a stochastic lookahead approach using stochastic programming with sample average approximation based on scenarios of the number of occupied infectious beds and infectious patients needing hospitalization. The lookahead approach produces high-quality decisions by measuring the impact of current decisions on future costs, such as costs for bed shortages, unused beds for infectious patients, and opening and closing rooms. Our simulation study applied to a COVID-19 scenario in the Netherlands, demonstrates that the stochastic lookahead approach outperforms a deterministic approach as well as other heuristic decision rules such as hospitals acting individually and implementing a pandemic unit, i.e., one hospital is designated to take all regional infectious patients until full. Our approach is very flexible and capable of tuning the model parameters to take into account the characteristics of future, yet unknown, pandemics, and supports sustaining regular care by minimizing the strain of infectious care on the available number of beds for regular care.
△ Less
Submitted 16 October, 2024; v1 submitted 27 November, 2023;
originally announced November 2023.
-
A Sampling-Based Method for Gittins Index Approximation
Authors:
Stef Baas,
Richard J. Boucherie,
Aleida Braaksma
Abstract:
A sampling-based method is introduced to approximate the Gittins index for a general family of alternative bandit processes. The approximation consists of a truncation of the optimization horizon and support for the immediate rewards, an optimal stopping value approximation, and a stochastic approximation procedure. Finite-time error bounds are given for the three approximations, leading to a proc…
▽ More
A sampling-based method is introduced to approximate the Gittins index for a general family of alternative bandit processes. The approximation consists of a truncation of the optimization horizon and support for the immediate rewards, an optimal stopping value approximation, and a stochastic approximation procedure. Finite-time error bounds are given for the three approximations, leading to a procedure to construct a confidence interval for the Gittins index using a finite number of Monte Carlo samples, as well as an epsilon-optimal policy for the Bayesian multi-armed bandit. Proofs are given for almost sure convergence and convergence in distribution for the sampling based Gittins index approximation. In a numerical study, the approximation quality of the proposed method is verified for the Bernoulli bandit and Gaussian bandit with known variance, and the method is shown to significantly outperform Thompson sampling and the Bayesian Upper Confidence Bound algorithms for a novel random effects multi-armed bandit.
△ Less
Submitted 21 July, 2023;
originally announced July 2023.
-
Bayesian Covariance Structure Modeling of Multi-Way Nested Data
Authors:
Stef Baas,
Richard J. Boucherie,
Jean-Paul Fox
Abstract:
A Bayesian multivariate model with a structured covariance matrix for multi-way nested data is proposed. This flexible modeling framework allows for positive and for negative associations among clustered observations, and generalizes the well-known dependence structure implied by random effects. A conjugate shifted-inverse gamma prior is proposed for the covariance parameters which ensures that th…
▽ More
A Bayesian multivariate model with a structured covariance matrix for multi-way nested data is proposed. This flexible modeling framework allows for positive and for negative associations among clustered observations, and generalizes the well-known dependence structure implied by random effects. A conjugate shifted-inverse gamma prior is proposed for the covariance parameters which ensures that the covariance matrix remains positive definite under posterior analysis. A numerically efficient Gibbs sampling procedure is defined for balanced nested designs, and is validated using two simulation studies. For a top-layer unbalanced nested design, the procedure requires an additional data augmentation step. The proposed data augmentation procedure facilitates sampling latent variables from (truncated) univariate normal distributions, and avoids numerical computation of the inverse of the structured covariance matrix. The Bayesian multivariate (linear transformation) model is applied to two-way nested interval-censored event times to analyze differences in adverse events between three groups of patients, who were randomly allocated to treatment with different stents (BIO-RESORT). The parameters of the structured covariance matrix represent unobserved heterogeneity in treatment effects and are examined to detect differential treatment effects.
△ Less
Submitted 25 January, 2022;
originally announced January 2022.
-
Ab initio Molecular Dynamical Investigation of the Finite Temperature Behavior of the Tetrahedral Au$_{19}$ and Au$_{20}$ Clusters
Authors:
Sailaja Krishnamurty,
Ghazal Shafai,
D. G. Kanhere,
B. Soulé de Bas,
M. J. Ford
Abstract:
Density functional molecular dynamics simulations have been carried out to understand the finite temperature behavior of Au$_{19}$ and Au$_{20}$ clusters. Au$_{20}$ has been reported to be a unique molecule having tetrahedral geometry, a large HOMO-LUMO energy gap and an atomic packing similar to that of the bulk gold (J. Li et al., Science, {\bf 299} 864, 2003). Our results show that the geomet…
▽ More
Density functional molecular dynamics simulations have been carried out to understand the finite temperature behavior of Au$_{19}$ and Au$_{20}$ clusters. Au$_{20}$ has been reported to be a unique molecule having tetrahedral geometry, a large HOMO-LUMO energy gap and an atomic packing similar to that of the bulk gold (J. Li et al., Science, {\bf 299} 864, 2003). Our results show that the geometry of Au$_{19}$ is exactly identical to that of Au$_{20}$ with one missing corner atom (called as vacancy). Surprisingly, our calculated heat capacities for this nearly identical pair of gold cluster exhibit dramatic differences. Au$_{20}$ undergoes a clear and distinct solid like to liquid like transition with a sharp peak in the heat capacity curve around 770 K. On the other hand, Au$_{19}$ has a broad and flat heat capacity curve with continuous melting transition. This continuous melting transition turns out to be a consequence of a process involving series of atomic rearrangements along the surface to fill in the missing corner atom. This results in a restricted diffusive motion of atoms along the surface of Au$_{19}$ between 650 K to 900 K during which the shape of the ground state geometry is retained. In contrast, the tetrahedral structure of Au$_{20}$ is destroyed around 800 K, and the cluster is clearly in a liquid like state above 1000 K. Thus, this work clearly demonstrates that (i) the gold clusters exhibit size sensitive variations in the heat capacity curves and (ii) the broad and continuous melting transition in a cluster, a feature which has so far been attributed to the disorder or absence of symmetry in the system, can also be a consequence of a defect (absence of a cap atom) in the structure.
△ Less
Submitted 12 December, 2006;
originally announced December 2006.
