-
Generalizing imaging biomarker repeatability studies using Bayesian inference: Applications in detecting heterogeneous treatment response in whole-body diffusion-weighted MRI of metastatic prostate cancer
Authors:
Matthew D Blackledge,
Konstantinos Zormpas-Petridis,
Ricardo Donners,
Antonio Candito,
David J Collins,
Johann de Bono,
Chris Parker,
Dow-Mu Koh,
Nina Tunariu
Abstract:
The assessment of imaging biomarkers is critical for advancing precision medicine and improving disease characterization. Despite the availability of methods to derive disease heterogeneity metrics in imaging studies, a robust framework for evaluating measurement uncertainty remains underdeveloped. To address this gap, we propose a novel Bayesian framework to assess the precision of disease hetero…
▽ More
The assessment of imaging biomarkers is critical for advancing precision medicine and improving disease characterization. Despite the availability of methods to derive disease heterogeneity metrics in imaging studies, a robust framework for evaluating measurement uncertainty remains underdeveloped. To address this gap, we propose a novel Bayesian framework to assess the precision of disease heterogeneity measures in biomarker studies.
Our approach extends traditional methods for evaluating biomarker precision by providing greater flexibility in statistical assumptions and enabling the analysis of biomarkers beyond univariate or multivariate normally-distributed variables. Using Hamiltonian Monte Carlo sampling, the framework supports both, for example, normally-distributed and Dirichlet-Multinomial distributed variables, enabling the derivation of posterior distributions for biomarker parameters under diverse model assumptions. Designed to be broadly applicable across various imaging modalities and biomarker types, the framework builds a foundation for generalizing reproducible and objective biomarker evaluation.
To demonstrate utility, we apply the framework to whole-body diffusion-weighted MRI (WBDWI) to assess heterogeneous therapeutic responses in metastatic bone disease. Specifically, we analyze data from two patient studies investigating treatments for metastatic castrate-resistant prostate cancer (mCRPC). Our results reveal an approximately 70% response rate among individual tumors across both studies, objectively characterizing differential responses to systemic therapies and validating the clinical relevance of the proposed methodology.
This Bayesian framework provides a powerful tool for advancing biomarker research across diverse imaging-based studies while offering valuable insights into specific clinical applications, such as mCRPC treatment response.
△ Less
Submitted 14 May, 2025;
originally announced May 2025.
-
Towards Understanding How Machines Can Learn Causal Overhypotheses
Authors:
Eliza Kosoy,
David M. Chan,
Adrian Liu,
Jasmine Collins,
Bryanna Kaufmann,
Sandy Han Huang,
Jessica B. Hamrick,
John Canny,
Nan Rosemary Ke,
Alison Gopnik
Abstract:
Recent work in machine learning and cognitive science has suggested that understanding causal information is essential to the development of intelligence. The extensive literature in cognitive science using the ``blicket detector'' environment shows that children are adept at many kinds of causal inference and learning. We propose to adapt that environment for machine learning agents. One of the k…
▽ More
Recent work in machine learning and cognitive science has suggested that understanding causal information is essential to the development of intelligence. The extensive literature in cognitive science using the ``blicket detector'' environment shows that children are adept at many kinds of causal inference and learning. We propose to adapt that environment for machine learning agents. One of the key challenges for current machine learning algorithms is modeling and understanding causal overhypotheses: transferable abstract hypotheses about sets of causal relationships. In contrast, even young children spontaneously learn and use causal overhypotheses. In this work, we present a new benchmark -- a flexible environment which allows for the evaluation of existing techniques under variable causal overhypotheses -- and demonstrate that many existing state-of-the-art methods have trouble generalizing in this environment. The code and resources for this benchmark are available at https://github.com/CannyLab/casual_overhypotheses.
△ Less
Submitted 16 June, 2022;
originally announced June 2022.
-
Comparing Weak- and Unsupervised Methods for Resonant Anomaly Detection
Authors:
Jack H. Collins,
Pablo MartÃn-Ramiro,
Benjamin Nachman,
David Shih
Abstract:
Anomaly detection techniques are growing in importance at the Large Hadron Collider (LHC), motivated by the increasing need to search for new physics in a model-agnostic way. In this work, we provide a detailed comparative study between a well-studied unsupervised method called the autoencoder (AE) and a weakly-supervised approach based on the Classification Without Labels (CWoLa) technique. We ex…
▽ More
Anomaly detection techniques are growing in importance at the Large Hadron Collider (LHC), motivated by the increasing need to search for new physics in a model-agnostic way. In this work, we provide a detailed comparative study between a well-studied unsupervised method called the autoencoder (AE) and a weakly-supervised approach based on the Classification Without Labels (CWoLa) technique. We examine the ability of the two methods to identify a new physics signal at different cross sections in a fully hadronic resonance search. By construction, the AE classification performance is independent of the amount of injected signal. In contrast, the CWoLa performance improves with increasing signal abundance. When integrating these approaches with a complete background estimate, we find that the two methods have complementary sensitivity. In particular, CWoLa is effective at finding diverse and moderately rare signals while the AE can provide sensitivity to very rare signals, but only with certain topologies. We therefore demonstrate that both techniques are complementary and can be used together for anomaly detection at the LHC.
△ Less
Submitted 5 April, 2021;
originally announced April 2021.
-
Estimating Road Network Accessibility during a Hurricane Evacuation: A Case Study of Hurricane Irma in Florida
Authors:
Yi-Jie Zhu,
Yujie Hu,
Jennifer M. Collins
Abstract:
Understanding the spatiotemporal road network accessibility during a hurricane evacuation, the level of ease of residents in an area in reaching evacuation destination sites through the road network, is a critical component of emergency management. While many studies have attempted to measure road accessibility (either in the scope of evacuation or beyond), few have considered both dynamic evacuat…
▽ More
Understanding the spatiotemporal road network accessibility during a hurricane evacuation, the level of ease of residents in an area in reaching evacuation destination sites through the road network, is a critical component of emergency management. While many studies have attempted to measure road accessibility (either in the scope of evacuation or beyond), few have considered both dynamic evacuation demand and characteristics of a hurricane. This study proposes a methodological framework to achieve this goal. In an interval of every six hours, the method first estimates the evacuation demand in terms of number of vehicles per household in each county subdivision by considering the hurricane's wind radius and track. The closest facility analysis is then employed to model evacuees' route choices towards the predefined evacuation destinations. The potential crowdedness index (PCI), a metric capturing the level of crowdedness of each road segment, is then computed by coupling the estimated evacuation demand and route choices. Finally, the road accessibility of each sub-county is measured by calculating the reciprocal of the sum of PCI values of corresponding roads connecting evacuees from the sub-county to the designated destinations. The method is applied to the entire state of Florida during Hurricane Irma in September 2017. Results show that I-75 and I-95 northbound have a high level of congestion, and sub-counties along the northbound I-95 suffer from the worst road accessibility. In addition, this research performs a sensitivity analysis for examining the impacts of different choices of behavioral response curves on accessibility results.
△ Less
Submitted 24 June, 2020;
originally announced June 2020.
-
Playing to Learn Better: Repeated Games for Adversarial Learning with Multiple Classifiers
Authors:
Prithviraj Dasgupta,
Joseph B. Collins,
Michael McCarrick
Abstract:
We consider the problem of prediction by a machine learning algorithm, called learner, within an adversarial learning setting. The learner's task is to correctly predict the class of data passed to it as a query. However, along with queries containing clean data, the learner could also receive malicious or adversarial queries from an adversary. The objective of the adversary is to evade the learne…
▽ More
We consider the problem of prediction by a machine learning algorithm, called learner, within an adversarial learning setting. The learner's task is to correctly predict the class of data passed to it as a query. However, along with queries containing clean data, the learner could also receive malicious or adversarial queries from an adversary. The objective of the adversary is to evade the learner's prediction mechanism by sending adversarial queries that result in erroneous class prediction by the learner, while the learner's objective is to reduce the incorrect prediction of these adversarial queries without degrading the prediction quality of clean queries. We propose a game theory-based technique called a Repeated Bayesian Sequential Game where the learner interacts repeatedly with a model of the adversary using self play to determine the distribution of adversarial versus clean queries. It then strategically selects a classifier from a set of pre-trained classifiers that balances the likelihood of correct prediction for the query along with reducing the costs to use the classifier. We have evaluated our proposed technique using clean and adversarial text data with deep neural network-based classifiers and shown that the learner can select an appropriate classifier that is commensurate with the query type (clean or adversarial) while remaining aware of the cost to use the classifier.
△ Less
Submitted 10 February, 2020;
originally announced February 2020.
-
A Survey of Game Theoretic Approaches for Adversarial Machine Learning in Cybersecurity Tasks
Authors:
Prithviraj Dasgupta,
Joseph B. Collins
Abstract:
Machine learning techniques are currently used extensively for automating various cybersecurity tasks. Most of these techniques utilize supervised learning algorithms that rely on training the algorithm to classify incoming data into different categories, using data encountered in the relevant domain. A critical vulnerability of these algorithms is that they are susceptible to adversarial attacks…
▽ More
Machine learning techniques are currently used extensively for automating various cybersecurity tasks. Most of these techniques utilize supervised learning algorithms that rely on training the algorithm to classify incoming data into different categories, using data encountered in the relevant domain. A critical vulnerability of these algorithms is that they are susceptible to adversarial attacks where a malicious entity called an adversary deliberately alters the training data to misguide the learning algorithm into making classification errors. Adversarial attacks could render the learning algorithm unsuitable to use and leave critical systems vulnerable to cybersecurity attacks. Our paper provides a detailed survey of the state-of-the-art techniques that are used to make a machine learning algorithm robust against adversarial attacks using the computational framework of game theory. We also discuss open problems and challenges and possible directions for further research that would make deep machine learning-based systems more robust and reliable for cybersecurity tasks.
△ Less
Submitted 4 December, 2019;
originally announced December 2019.
-
Accelerating Training of Deep Neural Networks with a Standardization Loss
Authors:
Jasmine Collins,
Johannes Balle,
Jonathon Shlens
Abstract:
A significant advance in accelerating neural network training has been the development of normalization methods, permitting the training of deep models both faster and with better accuracy. These advances come with practical challenges: for instance, batch normalization ties the prediction of individual examples with other examples within a batch, resulting in a network that is heavily dependent o…
▽ More
A significant advance in accelerating neural network training has been the development of normalization methods, permitting the training of deep models both faster and with better accuracy. These advances come with practical challenges: for instance, batch normalization ties the prediction of individual examples with other examples within a batch, resulting in a network that is heavily dependent on batch size. Layer normalization and group normalization are data-dependent and thus must be continually used, even at test-time. To address the issues that arise from using explicit normalization techniques, we propose to replace existing normalization methods with a simple, secondary objective loss that we term a standardization loss. This formulation is flexible and robust across different batch sizes and surprisingly, this secondary objective accelerates learning on the primary training objective. Because it is a training loss, it is simply removed at test-time, and no further effort is needed to maintain normalized activations. We find that a standardization loss accelerates training on both small- and large-scale image classification experiments, works with a variety of architectures, and is largely robust to training across different batch sizes.
△ Less
Submitted 3 March, 2019;
originally announced March 2019.
-
Capacity and Trainability in Recurrent Neural Networks
Authors:
Jasmine Collins,
Jascha Sohl-Dickstein,
David Sussillo
Abstract:
Two potential bottlenecks on the expressiveness of recurrent neural networks (RNNs) are their ability to store information about the task in their parameters, and to store information about the input history in their units. We show experimentally that all common RNN architectures achieve nearly the same per-task and per-unit capacity bounds with careful training, for a variety of tasks and stackin…
▽ More
Two potential bottlenecks on the expressiveness of recurrent neural networks (RNNs) are their ability to store information about the task in their parameters, and to store information about the input history in their units. We show experimentally that all common RNN architectures achieve nearly the same per-task and per-unit capacity bounds with careful training, for a variety of tasks and stacking depths. They can store an amount of task information which is linear in the number of parameters, and is approximately 5 bits per parameter. They can additionally store approximately one real number from their input history per hidden unit. We further find that for several tasks it is the per-task parameter capacity bound that determines performance. These results suggest that many previous results comparing RNN architectures are driven primarily by differences in training effectiveness, rather than differences in capacity. Supporting this observation, we compare training difficulty for several architectures, and show that vanilla RNNs are far more difficult to train, yet have slightly higher capacity. Finally, we propose two novel RNN architectures, one of which is easier to train than the LSTM or GRU for deeply stacked architectures.
△ Less
Submitted 3 March, 2017; v1 submitted 29 November, 2016;
originally announced November 2016.
-
Multi-Level Anomaly Detection on Time-Varying Graph Data
Authors:
Robert A. Bridges,
John Collins,
Erik M. Ferragut,
Jason Laska,
Blair D. Sullivan
Abstract:
This work presents a novel modeling and analysis framework for graph sequences which addresses the challenge of detecting and contextualizing anomalies in labelled, streaming graph data. We introduce a generalization of the BTER model of Seshadhri et al. by adding flexibility to community structure, and use this model to perform multi-scale graph anomaly detection. Specifically, probability models…
▽ More
This work presents a novel modeling and analysis framework for graph sequences which addresses the challenge of detecting and contextualizing anomalies in labelled, streaming graph data. We introduce a generalization of the BTER model of Seshadhri et al. by adding flexibility to community structure, and use this model to perform multi-scale graph anomaly detection. Specifically, probability models describing coarse subgraphs are built by aggregating probabilities at finer levels, and these closely related hierarchical models simultaneously detect deviations from expectation. This technique provides insight into a graph's structure and internal context that may shed light on a detected event. Additionally, this multi-scale analysis facilitates intuitive visualizations by allowing users to narrow focus from an anomalous graph to particular subgraphs or nodes causing the anomaly.
For evaluation, two hierarchical anomaly detectors are tested against a baseline Gaussian method on a series of sampled graphs. We demonstrate that our graph statistics-based approach outperforms both a distribution-based detector and the baseline in a labeled setting with community structure, and it accurately detects anomalies in synthetic and real-world datasets at the node, subgraph, and graph levels. To illustrate the accessibility of information made possible via this technique, the anomaly detector and an associated interactive visualization tool are tested on NCAA football data, where teams and conferences that moved within the league are identified with perfect recall, and precision greater than 0.786.
△ Less
Submitted 20 April, 2015; v1 submitted 16 October, 2014;
originally announced October 2014.
