-
MultiverSeg: Scalable Interactive Segmentation of Biomedical Imaging Datasets with In-Context Guidance
Authors:
Hallee E. Wong,
Jose Javier Gonzalez Ortiz,
John Guttag,
Adrian V. Dalca
Abstract:
Medical researchers and clinicians often need to perform novel segmentation tasks on a set of related images. Existing methods for segmenting a new dataset are either interactive, requiring substantial human effort for each image, or require an existing set of manually labeled images. We introduce a system, MultiverSeg, that enables practitioners to rapidly segment an entire new dataset without re…
▽ More
Medical researchers and clinicians often need to perform novel segmentation tasks on a set of related images. Existing methods for segmenting a new dataset are either interactive, requiring substantial human effort for each image, or require an existing set of manually labeled images. We introduce a system, MultiverSeg, that enables practitioners to rapidly segment an entire new dataset without requiring access to any existing labeled data from that task or domain. Along with the image to segment, the model takes user interactions such as clicks, bounding boxes or scribbles as input, and predicts a segmentation. As the user segments more images, those images and segmentations become additional inputs to the model, providing context. As the context set of labeled images grows, the number of interactions required to segment each new image decreases. We demonstrate that MultiverSeg enables users to interactively segment new datasets efficiently, by amortizing the number of interactions per image to achieve an accurate segmentation. Compared to using a state-of-the-art interactive segmentation method, using MultiverSeg reduced the total number of scribble steps by 53% and clicks by 36% to achieve 90% Dice on sets of images from unseen tasks. We release code and model weights at https://multiverseg.csail.mit.edu
△ Less
Submitted 19 December, 2024;
originally announced December 2024.
-
LoRA Learns Less and Forgets Less
Authors:
Dan Biderman,
Jacob Portes,
Jose Javier Gonzalez Ortiz,
Mansheej Paul,
Philip Greengard,
Connor Jennings,
Daniel King,
Sam Havens,
Vitaliy Chiley,
Jonathan Frankle,
Cody Blakeney,
John P. Cunningham
Abstract:
Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning method for large language models. LoRA saves memory by training only low rank perturbations to selected weight matrices. In this work, we compare the performance of LoRA and full finetuning on two target domains, programming and mathematics. We consider both the instruction finetuning (approximately 100K prompt-response pai…
▽ More
Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning method for large language models. LoRA saves memory by training only low rank perturbations to selected weight matrices. In this work, we compare the performance of LoRA and full finetuning on two target domains, programming and mathematics. We consider both the instruction finetuning (approximately 100K prompt-response pairs) and continued pretraining (20B unstructured tokens) data regimes. Our results show that, in the standard low-rank settings, LoRA substantially underperforms full finetuning. Nevertheless, LoRA better maintains the base model's performance on tasks outside the target domain. We show that LoRA mitigates forgetting more than common regularization techniques such as weight decay and dropout; it also helps maintain more diverse generations. Finally, we show that full finetuning learns perturbations with a rank that is 10-100X greater than typical LoRA configurations, possibly explaining some of the reported gaps. We conclude by proposing best practices for finetuning with LoRA.
△ Less
Submitted 20 September, 2024; v1 submitted 15 May, 2024;
originally announced May 2024.
-
Tyche: Stochastic In-Context Learning for Medical Image Segmentation
Authors:
Marianne Rakic,
Hallee E. Wong,
Jose Javier Gonzalez Ortiz,
Beth Cimini,
John Guttag,
Adrian V. Dalca
Abstract:
Existing learning-based solutions to medical image segmentation have two important shortcomings. First, for most new segmentation task, a new model has to be trained or fine-tuned. This requires extensive resources and machine learning expertise, and is therefore often infeasible for medical researchers and clinicians. Second, most existing segmentation methods produce a single deterministic segme…
▽ More
Existing learning-based solutions to medical image segmentation have two important shortcomings. First, for most new segmentation task, a new model has to be trained or fine-tuned. This requires extensive resources and machine learning expertise, and is therefore often infeasible for medical researchers and clinicians. Second, most existing segmentation methods produce a single deterministic segmentation mask for a given image. In practice however, there is often considerable uncertainty about what constitutes the correct segmentation, and different expert annotators will often segment the same image differently. We tackle both of these problems with Tyche, a model that uses a context set to generate stochastic predictions for previously unseen tasks without the need to retrain. Tyche differs from other in-context segmentation methods in two important ways. (1) We introduce a novel convolution block architecture that enables interactions among predictions. (2) We introduce in-context test-time augmentation, a new mechanism to provide prediction stochasticity. When combined with appropriate model design and loss functions, Tyche can predict a set of plausible diverse segmentation candidates for new or unseen medical images and segmentation tasks without the need to retrain.
△ Less
Submitted 24 January, 2024;
originally announced January 2024.
-
Magnitude Invariant Parametrizations Improve Hypernetwork Learning
Authors:
Jose Javier Gonzalez Ortiz,
John Guttag,
Adrian Dalca
Abstract:
Hypernetworks, neural networks that predict the parameters of another neural network, are powerful models that have been successfully used in diverse applications from image generation to multi-task learning. Unfortunately, existing hypernetworks are often challenging to train. Training typically converges far more slowly than for non-hypernetwork models, and the rate of convergence can be very se…
▽ More
Hypernetworks, neural networks that predict the parameters of another neural network, are powerful models that have been successfully used in diverse applications from image generation to multi-task learning. Unfortunately, existing hypernetworks are often challenging to train. Training typically converges far more slowly than for non-hypernetwork models, and the rate of convergence can be very sensitive to hyperparameter choices. In this work, we identify a fundamental and previously unidentified problem that contributes to the challenge of training hypernetworks: a magnitude proportionality between the inputs and outputs of the hypernetwork. We demonstrate both analytically and empirically that this can lead to unstable optimization, thereby slowing down convergence, and sometimes even preventing any learning. We present a simple solution to this problem using a revised hypernetwork formulation that we call Magnitude Invariant Parametrizations (MIP). We demonstrate the proposed solution on several hypernetwork tasks, where it consistently stabilizes training and achieves faster convergence. Furthermore, we perform a comprehensive ablation study including choices of activation function, normalization strategies, input dimensionality, and hypernetwork architecture; and find that MIP improves training in all scenarios. We provide easy-to-use code that can turn existing networks into MIP-based hypernetworks.
△ Less
Submitted 29 June, 2023; v1 submitted 15 April, 2023;
originally announced April 2023.
-
UniverSeg: Universal Medical Image Segmentation
Authors:
Victor Ion Butoi,
Jose Javier Gonzalez Ortiz,
Tianyu Ma,
Mert R. Sabuncu,
John Guttag,
Adrian V. Dalca
Abstract:
While deep learning models have become the predominant method for medical image segmentation, they are typically not capable of generalizing to unseen segmentation tasks involving new anatomies, image modalities, or labels. Given a new segmentation task, researchers generally have to train or fine-tune models, which is time-consuming and poses a substantial barrier for clinical researchers, who of…
▽ More
While deep learning models have become the predominant method for medical image segmentation, they are typically not capable of generalizing to unseen segmentation tasks involving new anatomies, image modalities, or labels. Given a new segmentation task, researchers generally have to train or fine-tune models, which is time-consuming and poses a substantial barrier for clinical researchers, who often lack the resources and expertise to train neural networks. We present UniverSeg, a method for solving unseen medical segmentation tasks without additional training. Given a query image and example set of image-label pairs that define a new segmentation task, UniverSeg employs a new Cross-Block mechanism to produce accurate segmentation maps without the need for additional training. To achieve generalization to new tasks, we have gathered and standardized a collection of 53 open-access medical segmentation datasets with over 22,000 scans, which we refer to as MegaMedical. We used this collection to train UniverSeg on a diverse set of anatomies and imaging modalities. We demonstrate that UniverSeg substantially outperforms several related methods on unseen tasks, and thoroughly analyze and draw insights about important aspects of the proposed system. The UniverSeg source code and model weights are freely available at https://universeg.csail.mit.edu
△ Less
Submitted 12 April, 2023;
originally announced April 2023.
-
Scale-Space Hypernetworks for Efficient Biomedical Imaging
Authors:
Jose Javier Gonzalez Ortiz,
John Guttag,
Adrian Dalca
Abstract:
Convolutional Neural Networks (CNNs) are the predominant model used for a variety of medical image analysis tasks. At inference time, these models are computationally intensive, especially with volumetric data. In principle, it is possible to trade accuracy for computational efficiency by manipulating the rescaling factor in the downsample and upsample layers of CNN architectures. However, properl…
▽ More
Convolutional Neural Networks (CNNs) are the predominant model used for a variety of medical image analysis tasks. At inference time, these models are computationally intensive, especially with volumetric data. In principle, it is possible to trade accuracy for computational efficiency by manipulating the rescaling factor in the downsample and upsample layers of CNN architectures. However, properly exploring the accuracy-efficiency trade-off is prohibitively expensive with existing models. To address this, we introduce Scale-Space HyperNetworks (SSHN), a method that learns a spectrum of CNNs with varying internal rescaling factors. A single SSHN characterizes an entire Pareto accuracy-efficiency curve of models that match, and occasionally surpass, the outcomes of training many separate networks with fixed rescaling factors. We demonstrate the proposed approach in several medical image analysis applications, comparing SSHN against strategies with both fixed and dynamic rescaling factors. We find that SSHN consistently provides a better accuracy-efficiency trade-off at a fraction of the training cost. Trained SSHNs enable the user to quickly choose a rescaling factor that appropriately balances accuracy and computational efficiency for their particular needs at inference.
△ Less
Submitted 29 June, 2023; v1 submitted 11 April, 2023;
originally announced April 2023.
-
Human-Robot Commensality: Bite Timing Prediction for Robot-Assisted Feeding in Groups
Authors:
Jan Ondras,
Abrar Anwar,
Tong Wu,
Fanjun Bu,
Malte Jung,
Jorge Jose Ortiz,
Tapomayukh Bhattacharjee
Abstract:
We develop data-driven models to predict when a robot should feed during social dining scenarios. Being able to eat independently with friends and family is considered one of the most memorable and important activities for people with mobility limitations. While existing robotic systems for feeding people with mobility limitations focus on solitary dining, commensality, the act of eating together,…
▽ More
We develop data-driven models to predict when a robot should feed during social dining scenarios. Being able to eat independently with friends and family is considered one of the most memorable and important activities for people with mobility limitations. While existing robotic systems for feeding people with mobility limitations focus on solitary dining, commensality, the act of eating together, is often the practice of choice. Sharing meals with others introduces the problem of socially appropriate bite timing for a robot, i.e. the appropriate timing for the robot to feed without disrupting the social dynamics of a shared meal. Our key insight is that bite timing strategies that take into account the delicate balance of social cues can lead to seamless interactions during robot-assisted feeding in a social dining scenario. We approach this problem by collecting a Human-Human Commensality Dataset (HHCD) containing 30 groups of three people eating together. We use this dataset to analyze human-human commensality behaviors and develop bite timing prediction models in social dining scenarios. We also transfer these models to human-robot commensality scenarios. Our user studies show that prediction improves when our algorithm uses multimodal social signaling cues between diners to model bite timing. The HHCD dataset, videos of user studies, and code are available at https://emprise.cs.cornell.edu/hrcom/
△ Less
Submitted 16 November, 2022; v1 submitted 7 July, 2022;
originally announced July 2022.
-
Trade-offs of Local SGD at Scale: An Empirical Study
Authors:
Jose Javier Gonzalez Ortiz,
Jonathan Frankle,
Mike Rabbat,
Ari Morcos,
Nicolas Ballas
Abstract:
As datasets and models become increasingly large, distributed training has become a necessary component to allow deep neural networks to train in reasonable amounts of time. However, distributed training can have substantial communication overhead that hinders its scalability. One strategy for reducing this overhead is to perform multiple unsynchronized SGD steps independently on each worker betwe…
▽ More
As datasets and models become increasingly large, distributed training has become a necessary component to allow deep neural networks to train in reasonable amounts of time. However, distributed training can have substantial communication overhead that hinders its scalability. One strategy for reducing this overhead is to perform multiple unsynchronized SGD steps independently on each worker between synchronization steps, a technique known as local SGD. We conduct a comprehensive empirical study of local SGD and related methods on a large-scale image classification task. We find that performing local SGD comes at a price: lower communication costs (and thereby faster training) are accompanied by lower accuracy. This finding is in contrast from the smaller-scale experiments in prior work, suggesting that local SGD encounters challenges at scale. We further show that incorporating the slow momentum framework of Wang et al. (2020) consistently improves accuracy without requiring additional communication, hinting at future directions for potentially escaping this trade-off.
△ Less
Submitted 15 October, 2021;
originally announced October 2021.
-
What is the State of Neural Network Pruning?
Authors:
Davis Blalock,
Jose Javier Gonzalez Ortiz,
Jonathan Frankle,
John Guttag
Abstract:
Neural network pruning---the task of reducing the size of a network by removing parameters---has been the subject of a great deal of work in recent years. We provide a meta-analysis of the literature, including an overview of approaches to pruning and consistent findings in the literature. After aggregating results across 81 papers and pruning hundreds of models in controlled conditions, our clear…
▽ More
Neural network pruning---the task of reducing the size of a network by removing parameters---has been the subject of a great deal of work in recent years. We provide a meta-analysis of the literature, including an overview of approaches to pruning and consistent findings in the literature. After aggregating results across 81 papers and pruning hundreds of models in controlled conditions, our clearest finding is that the community suffers from a lack of standardized benchmarks and metrics. This deficiency is substantial enough that it is hard to compare pruning techniques to one another or determine how much progress the field has made over the past three decades. To address this situation, we identify issues with current practices, suggest concrete remedies, and introduce ShrinkBench, an open-source framework to facilitate standardized evaluations of pruning methods. We use ShrinkBench to compare various pruning techniques and show that its comprehensive evaluation can prevent common pitfalls when comparing pruning methods.
△ Less
Submitted 6 March, 2020;
originally announced March 2020.
-
Image segmentation of liver stage malaria infection with spatial uncertainty sampling
Authors:
Ava P. Soleimany,
Harini Suresh,
Jose Javier Gonzalez Ortiz,
Divya Shanmugam,
Nil Gural,
John Guttag,
Sangeeta N. Bhatia
Abstract:
Global eradication of malaria depends on the development of drugs effective against the silent, yet obligate liver stage of the disease. The gold standard in drug development remains microscopic imaging of liver stage parasites in in vitro cell culture models. Image analysis presents a major bottleneck in this pipeline since the parasite has significant variability in size, shape, and density in t…
▽ More
Global eradication of malaria depends on the development of drugs effective against the silent, yet obligate liver stage of the disease. The gold standard in drug development remains microscopic imaging of liver stage parasites in in vitro cell culture models. Image analysis presents a major bottleneck in this pipeline since the parasite has significant variability in size, shape, and density in these models. As with other highly variable datasets, traditional segmentation models have poor generalizability as they rely on hand-crafted features; thus, manual annotation of liver stage malaria images remains standard. To address this need, we develop a convolutional neural network architecture that utilizes spatial dropout sampling for parasite segmentation and epistemic uncertainty estimation in images of liver stage malaria. Our pipeline produces high-precision segmentations nearly identical to expert annotations, generalizes well on a diverse dataset of liver stage malaria parasites, and promotes independence between learned feature maps to model the uncertainty of generated predictions.
△ Less
Submitted 30 November, 2019;
originally announced December 2019.
-
A Simple Power Analysis Attack on the Twofish Key Schedule
Authors:
Jose Javier Gonzalez Ortiz,
Kevin J. Compton
Abstract:
This paper introduces an SPA power attack on the 8-bit implementation of the Twofish block cipher. The attack is able to unequivocally recover the secret key even under substantial amounts of error. An initial algorithm is described using exhaustive search on error free data. An error resistant algorithm is later described. It employs several threshold preprocessing stages followed by a combined a…
▽ More
This paper introduces an SPA power attack on the 8-bit implementation of the Twofish block cipher. The attack is able to unequivocally recover the secret key even under substantial amounts of error. An initial algorithm is described using exhaustive search on error free data. An error resistant algorithm is later described. It employs several threshold preprocessing stages followed by a combined approach of least mean squares and an optimized Hamming mask search. Further analysis of 32 and 64-bit Twofish implementations reveals that they are similarly vulnerable to the described SPA attack.
△ Less
Submitted 21 November, 2016;
originally announced November 2016.
-
Scalable design of an IMS cross-flow micro-generator/ion detector
Authors:
Juan J. Ortiz,
Guillermo P. Ortiz,
Christian Nigri,
Carlos Lasorsa
Abstract:
Ion-mobility spectrometry (IMS) is an analytical technique used to separate and identify ionized gas molecules based on their mobility in a carrier buffer gas. Such methods come in a large variety of versions that currently allow ion identification at and above the millimeter scale. Here, we present a design for a cross-flow-IMS method able to generate and detect ions at the sub-millimeter scale.…
▽ More
Ion-mobility spectrometry (IMS) is an analytical technique used to separate and identify ionized gas molecules based on their mobility in a carrier buffer gas. Such methods come in a large variety of versions that currently allow ion identification at and above the millimeter scale. Here, we present a design for a cross-flow-IMS method able to generate and detect ions at the sub-millimeter scale. We propose a novel ion focusing strategy and tested it in a prototype device using Nitrogen as a sample gas, and also with simulations using four different sample gases. By introducing an original lobular ion generation localized to a few ten of microns and substantially simplifying the design, our device is able to keep constant laminar flow conditions for high flow rates. In this way, it avoids the turbulences in the gas flow, which would occur in other ion-focusing cross-flow methods limiting their performance at the sub-millimeter scale. Scalability of the proposed design can contribute to improve resolving power and resolution of currently available cross-flow methods.
△ Less
Submitted 14 February, 2013; v1 submitted 30 July, 2012;
originally announced July 2012.
-
Development of an IMS Type Device for Volatile Organic Compunds Detection: Simulation and Comparision of the Ion Distributions
Authors:
Guillermo P. Ortiz,
Carlos A. Rinaldi,
Norberto G. Boggio,
Juan Vorobioff,
Juan J. Ortiz,
Sergio Gómez,
Gustavo A. Aucar,
Alberto Lamagna,
Alfredo Boselli
Abstract:
Ion Mobility Spectrometry (IMS) is a well-known, sensitive and rapid technique to detect dangerous organic compounds. We propose a system in which a crown type discharge generates a ionic flow that is swept towards an array of collectors by a transverse electric field. The ions are separated as they enter the cell according to their mobility. Thus, the distribution of the charge deposited at the…
▽ More
Ion Mobility Spectrometry (IMS) is a well-known, sensitive and rapid technique to detect dangerous organic compounds. We propose a system in which a crown type discharge generates a ionic flow that is swept towards an array of collectors by a transverse electric field. The ions are separated as they enter the cell according to their mobility. Thus, the distribution of the charge deposited at the detector assembly constitutes a {\em fingerprint} for each organic compound. Simulations of our cell and experiments were performed for small amounts of acetone, ethanol and toluene. The dependence on the cell parameters of the current and charge versus time of flight was analyzed. Our simulation reproduces only qualitatively the experimental results. However, a PCA statistical analysis of the results obtained by simulation of the proposed design shows that the fingerprint is useful for a clear identification of such compounds.
△ Less
Submitted 22 March, 2009; v1 submitted 6 February, 2009;
originally announced February 2009.