-
One Representation to Rule Them All: Identifying Out-of-Support Examples in Few-shot Learning with Generic Representations
Authors:
Henry Kvinge,
Scott Howland,
Nico Courts,
Lauren A. Phillips,
John Buckheit,
Zachary New,
Elliott Skomski,
Jung H. Lee,
Sandeep Tiwari,
Jessica Hibler,
Courtney D. Corley,
Nathan O. Hodas
Abstract:
The field of few-shot learning has made remarkable strides in developing powerful models that can operate in the small data regime. Nearly all of these methods assume every unlabeled instance encountered will belong to a handful of known classes for which one has examples. This can be problematic for real-world use cases where one routinely finds 'none-of-the-above' examples. In this paper we desc…
▽ More
The field of few-shot learning has made remarkable strides in developing powerful models that can operate in the small data regime. Nearly all of these methods assume every unlabeled instance encountered will belong to a handful of known classes for which one has examples. This can be problematic for real-world use cases where one routinely finds 'none-of-the-above' examples. In this paper we describe this challenge of identifying what we term 'out-of-support' (OOS) examples. We describe how this problem is subtly different from out-of-distribution detection and describe a new method of identifying OOS examples within the Prototypical Networks framework using a fixed point which we call the generic representation. We show that our method outperforms other existing approaches in the literature as well as other approaches that we propose in this paper. Finally, we investigate how the use of such a generic point affects the geometry of a model's feature space.
△ Less
Submitted 2 June, 2021;
originally announced June 2021.
-
Fuzzy Simplicial Networks: A Topology-Inspired Model to Improve Task Generalization in Few-shot Learning
Authors:
Henry Kvinge,
Zachary New,
Nico Courts,
Jung H. Lee,
Lauren A. Phillips,
Courtney D. Corley,
Aaron Tuor,
Andrew Avila,
Nathan O. Hodas
Abstract:
Deep learning has shown great success in settings with massive amounts of data but has struggled when data is limited. Few-shot learning algorithms, which seek to address this limitation, are designed to generalize well to new tasks with limited data. Typically, models are evaluated on unseen classes and datasets that are defined by the same fundamental task as they are trained for (e.g. category…
▽ More
Deep learning has shown great success in settings with massive amounts of data but has struggled when data is limited. Few-shot learning algorithms, which seek to address this limitation, are designed to generalize well to new tasks with limited data. Typically, models are evaluated on unseen classes and datasets that are defined by the same fundamental task as they are trained for (e.g. category membership). One can also ask how well a model can generalize to fundamentally different tasks within a fixed dataset (for example: moving from category membership to tasks that involve detecting object orientation or quantity). To formalize this kind of shift we define a notion of "independence of tasks" and identify three new sets of labels for established computer vision datasets that test a model's ability to generalize to tasks which draw on orthogonal attributes in the data. We use these datasets to investigate the failure modes of metric-based few-shot models. Based on our findings, we introduce a new few-shot model called Fuzzy Simplicial Networks (FSN) which leverages a construction from topology to more flexibly represent each class from limited data. In particular, FSN models can not only form multiple representations for a given class but can also begin to capture the low-dimensional structure which characterizes class manifolds in the encoded space of deep networks. We show that FSN outperforms state-of-the-art models on the challenging tasks we introduce in this paper while remaining competitive on standard few-shot benchmarks.
△ Less
Submitted 23 September, 2020;
originally announced September 2020.
-
Mutual information for fitting deep nonlinear models
Authors:
Jacob S. Hunter,
Nathan O. Hodas
Abstract:
Deep nonlinear models pose a challenge for fitting parameters due to lack of knowledge of the hidden layer and the potentially non-affine relation of the initial and observed layers. In the present work we investigate the use of information theoretic measures such as mutual information and Kullback-Leibler (KL) divergence as objective functions for fitting such models without knowledge of the hidd…
▽ More
Deep nonlinear models pose a challenge for fitting parameters due to lack of knowledge of the hidden layer and the potentially non-affine relation of the initial and observed layers. In the present work we investigate the use of information theoretic measures such as mutual information and Kullback-Leibler (KL) divergence as objective functions for fitting such models without knowledge of the hidden layer. We investigate one model as a proof of concept and one application of cogntive performance. We further investigate the use of optimizers with these methods. Mutual information is largely successful as an objective, depending on the parameters. KL divergence is found to be similarly succesful, given some knowledge of the statistics of the hidden layer.
△ Less
Submitted 17 December, 2016;
originally announced December 2016.
-
The Quality of Oscillations in Overdamped Networks
Authors:
Nathan O. Hodas
Abstract:
The second law of thermodynamics implies that no macroscopic system may oscillate indefinitely without consuming energy. The question of the number of possible oscillations and the coherent quality of these oscillations remain unanswered. This paper proves the upper-bounds on the number and quality of such oscillations when the system in question is homogeneously driven and has a discrete network…
▽ More
The second law of thermodynamics implies that no macroscopic system may oscillate indefinitely without consuming energy. The question of the number of possible oscillations and the coherent quality of these oscillations remain unanswered. This paper proves the upper-bounds on the number and quality of such oscillations when the system in question is homogeneously driven and has a discrete network of states. In a closed system, the maximum number of oscillations is bounded by the number of states in the network. In open systems, the size of the network bounds the quality factor of oscillation. This work also explores how the quality factor of macrostate oscillations, such as would be observed in chemical reactions, are bounded by the smallest equivalent loop of the network, not the size of the entire system. The consequences of this limit are explored in the context of chemical clocks and limit cycles.
△ Less
Submitted 8 August, 2011; v1 submitted 1 June, 2010;
originally announced June 2010.
