-
Oops, I Sampled it Again: Reinterpreting Confidence Intervals in Few-Shot Learning
Authors:
Raphael Lafargue,
Luke Smith,
Franck Vermet,
Mathias Löwe,
Ian Reid,
Vincent Gripon,
Jack Valmadre
Abstract:
The predominant method for computing confidence intervals (CI) in few-shot learning (FSL) is based on sampling the tasks with replacement, i.e.\ allowing the same samples to appear in multiple tasks. This makes the CI misleading in that it takes into account the randomness of the sampler but not the data itself. To quantify the extent of this problem, we conduct a comparative analysis between CIs…
▽ More
The predominant method for computing confidence intervals (CI) in few-shot learning (FSL) is based on sampling the tasks with replacement, i.e.\ allowing the same samples to appear in multiple tasks. This makes the CI misleading in that it takes into account the randomness of the sampler but not the data itself. To quantify the extent of this problem, we conduct a comparative analysis between CIs computed with and without replacement. These reveal a notable underestimation by the predominant method. This observation calls for a reevaluation of how we interpret confidence intervals and the resulting conclusions in FSL comparative studies. Our research demonstrates that the use of paired tests can partially address this issue. Additionally, we explore methods to further reduce the (size of the) CI by strategically sampling tasks of a specific size. We also introduce a new optimized benchmark, which can be accessed at https://github.com/RafLaf/FSL-benchmark-again
△ Less
Submitted 6 September, 2024; v1 submitted 4 September, 2024;
originally announced September 2024.
-
Collaborative Insurance Sustainability and Network Structure
Authors:
Arthur Charpentier,
Lariosse Kouakou,
Matthias Löwe,
Philipp Ratz,
Franck Vermet
Abstract:
The peer-to-peer (P2P) economy has been growing with the advent of the Internet, with well known brands such as Uber or Airbnb being examples thereof. In the insurance sector the approach is still in its infancy, but some companies have started to explore P2P-based collaborative insurance products (eg. Lemonade in the U.S. or Inspeer in France). The actuarial literature only recently started to co…
▽ More
The peer-to-peer (P2P) economy has been growing with the advent of the Internet, with well known brands such as Uber or Airbnb being examples thereof. In the insurance sector the approach is still in its infancy, but some companies have started to explore P2P-based collaborative insurance products (eg. Lemonade in the U.S. or Inspeer in France). The actuarial literature only recently started to consider those risk sharing mechanisms, as in Denuit and Robert (2021) or Feng et al. (2021). In this paper, describe and analyse such a P2P product, with some reciprocal risk sharing contracts. Here, we consider the case where policyholders still have an insurance contract, but the first self-insurance layer, below the deductible, can be shared with friends. We study the impact of the shape of the network (through the distribution of degrees) on the risk reduction. We consider also some optimal setting of the reciprocal commitments, and discuss the introduction of contracts with friends of friends to mitigate some possible drawbacks of having people without enough connections to exchange risks.
△ Less
Submitted 12 September, 2022; v1 submitted 5 July, 2021;
originally announced July 2021.
-
Dealing with Adversarial Player Strategies in the Neural Network Game iNNk through Ensemble Learning
Authors:
Mathias Löwe,
Jennifer Villareale,
Evan Freed,
Aleksanteri Sladek,
Jichen Zhu,
Sebastian Risi
Abstract:
Applying neural network (NN) methods in games can lead to various new and exciting game dynamics not previously possible. However, they also lead to new challenges such as the lack of large, clean datasets, varying player skill levels, and changing gameplay strategies. In this paper, we focus on the adversarial player strategy aspect in the game iNNk, in which players try to communicate secret cod…
▽ More
Applying neural network (NN) methods in games can lead to various new and exciting game dynamics not previously possible. However, they also lead to new challenges such as the lack of large, clean datasets, varying player skill levels, and changing gameplay strategies. In this paper, we focus on the adversarial player strategy aspect in the game iNNk, in which players try to communicate secret code words through drawings with the goal of not being deciphered by a NN. Some strategies exploit weaknesses in the NN that consistently trick it into making incorrect classifications, leading to unbalanced gameplay. We present a method that combines transfer learning and ensemble methods to obtain a data-efficient adaptation to these strategies. This combination significantly outperforms the baseline NN across all adversarial player strategies despite only being trained on a limited set of adversarial examples. We expect the methods developed in this paper to be useful for the rapidly growing field of NN-based games, which will require new approaches to deal with unforeseen player creativity.
△ Less
Submitted 5 July, 2021;
originally announced July 2021.
-
A Generalized Framework for Measuring Pedestrian Accessibility around the World Using Open Data
Authors:
Shiqin Liu,
Carl Higgs,
Jonathan Arundel,
Geoff Boeing,
Nicholas Cerdera,
David Moctezuma,
Ester Cerin,
Deepti Adlakha,
Melanie Lowe,
Billie Giles-Corti
Abstract:
Pedestrian accessibility is an important factor in urban transport and land use policy and critical for creating healthy, sustainable cities. Developing and evaluating indicators measuring inequalities in pedestrian accessibility can help planners and policymakers benchmark and monitor the progress of city planning interventions. However, measuring and assessing indicators of urban design and tran…
▽ More
Pedestrian accessibility is an important factor in urban transport and land use policy and critical for creating healthy, sustainable cities. Developing and evaluating indicators measuring inequalities in pedestrian accessibility can help planners and policymakers benchmark and monitor the progress of city planning interventions. However, measuring and assessing indicators of urban design and transport features at high resolution worldwide to enable city comparisons is challenging due to limited availability of official, high quality, and comparable spatial data, as well as spatial analysis tools offering customizable frameworks for indicator construction and analysis. To address these challenges, this study develops an open source software framework to construct pedestrian accessibility indicators for cities using open and consistent data. It presents a generalized method to consistently measure pedestrian accessibility at high resolution and spatially aggregated scale, to allow for both within- and between-city analyses. The open source and open data methods developed in this study can be extended to other cities worldwide to support local planning and policymaking. The software is made publicly available for reuse in an open repository.
△ Less
Submitted 18 May, 2021;
originally announced May 2021.
-
Rapid Risk Minimization with Bayesian Models Through Deep Learning Approximation
Authors:
Mathias Löwe,
Per Lunnemann Hansen,
Sebastian Risi
Abstract:
We introduce a novel combination of Bayesian Models (BMs) and Neural Networks (NNs) for making predictions with a minimum expected risk. Our approach combines the best of both worlds, the data efficiency and interpretability of a BM with the speed of a NN. For a BM, making predictions with the lowest expected loss requires integrating over the posterior distribution. When exact inference of the po…
▽ More
We introduce a novel combination of Bayesian Models (BMs) and Neural Networks (NNs) for making predictions with a minimum expected risk. Our approach combines the best of both worlds, the data efficiency and interpretability of a BM with the speed of a NN. For a BM, making predictions with the lowest expected loss requires integrating over the posterior distribution. When exact inference of the posterior predictive distribution is intractable, approximation methods are typically applied, e.g. Monte Carlo (MC) simulation. For MC, the variance of the estimator decreases with the number of samples - but at the expense of increased computational cost. Our approach removes the need for iterative MC simulation on the CPU at prediction time. In brief, it works by fitting a NN to synthetic data generated using the BM. In a single feed-forward pass, the NN gives a set of point-wise approximations to the BM's posterior predictive distribution for a given observation. We achieve risk minimized predictions significantly faster than standard methods with a negligible loss on the test dataset. We combine this approach with Active Learning to minimize the amount of data required for fitting the NN. This is done by iteratively labeling more data in regions with high predictive uncertainty of the NN.
△ Less
Submitted 5 May, 2021; v1 submitted 29 March, 2021;
originally announced March 2021.
-
Player-AI Interaction: What Neural Network Games Reveal About AI as Play
Authors:
Jichen Zhu,
Jennifer Villareale,
Nithesh Javvaji,
Sebastian Risi,
Mathias Löwe,
Rush Weigelt,
Casper Harteveld
Abstract:
The advent of artificial intelligence (AI) and machine learning (ML) bring human-AI interaction to the forefront of HCI research. This paper argues that games are an ideal domain for studying and experimenting with how humans interact with AI. Through a systematic survey of neural network games (n = 38), we identified the dominant interaction metaphors and AI interaction patterns in these games. I…
▽ More
The advent of artificial intelligence (AI) and machine learning (ML) bring human-AI interaction to the forefront of HCI research. This paper argues that games are an ideal domain for studying and experimenting with how humans interact with AI. Through a systematic survey of neural network games (n = 38), we identified the dominant interaction metaphors and AI interaction patterns in these games. In addition, we applied existing human-AI interaction guidelines to further shed light on player-AI interaction in the context of AI-infused systems. Our core finding is that AI as play can expand current notions of human-AI interaction, which are predominantly productivity-based. In particular, our work suggests that game and UX designers should consider flow to structure the learning curve of human-AI interaction, incorporate discovery-based learning to play around with the AI and observe the consequences, and offer users an invitation to play to explore new forms of human-AI interaction.
△ Less
Submitted 18 January, 2021; v1 submitted 15 January, 2021;
originally announced January 2021.
-
Some Remarks on Replicated Simulated Annealing
Authors:
Vincent Gripon,
Matthias Löwe,
Franck Vermet
Abstract:
Recently authors have introduced the idea of training discrete weights neural networks using a mix between classical simulated annealing and a replica ansatz known from the statistical physics literature. Among other points, they claim their method is able to find robust configurations. In this paper, we analyze this so-called "replicated simulated annealing" algorithm. In particular, we explicit…
▽ More
Recently authors have introduced the idea of training discrete weights neural networks using a mix between classical simulated annealing and a replica ansatz known from the statistical physics literature. Among other points, they claim their method is able to find robust configurations. In this paper, we analyze this so-called "replicated simulated annealing" algorithm. In particular, we explicit criteria to guarantee its convergence, and study when it successfully samples from configurations. We also perform experiments using synthetic and real data bases.
△ Less
Submitted 2 December, 2020; v1 submitted 30 September, 2020;
originally announced September 2020.
-
iNNk: A Multi-Player Game to Deceive a Neural Network
Authors:
Jennifer Villareale,
Ana Acosta-Ruiz,
Samuel Arcaro,
Thomas Fox,
Evan Freed,
Robert Gray,
Mathias Löwe,
Panote Nuchprayoon,
Aleksanteri Sladek,
Rush Weigelt,
Yifu Li,
Sebastian Risi,
Jichen Zhu
Abstract:
This paper presents iNNK, a multiplayer drawing game where human players team up against an NN. The players need to successfully communicate a secret code word to each other through drawings, without being deciphered by the NN. With this game, we aim to foster a playful environment where players can, in a small way, go from passive consumers of NN applications to creative thinkers and critical cha…
▽ More
This paper presents iNNK, a multiplayer drawing game where human players team up against an NN. The players need to successfully communicate a secret code word to each other through drawings, without being deciphered by the NN. With this game, we aim to foster a playful environment where players can, in a small way, go from passive consumers of NN applications to creative thinkers and critical challengers.
△ Less
Submitted 15 January, 2021; v1 submitted 17 July, 2020;
originally announced July 2020.
-
Towards an Intrinsic Definition of Robustness for a Classifier
Authors:
Théo Giraudon,
Vincent Gripon,
Matthias Löwe,
Franck Vermet
Abstract:
The robustness of classifiers has become a question of paramount importance in the past few years. Indeed, it has been shown that state-of-the-art deep learning architectures can easily be fooled with imperceptible changes to their inputs. Therefore, finding good measures of robustness of a trained classifier is a key issue in the field. In this paper, we point out that averaging the radius of rob…
▽ More
The robustness of classifiers has become a question of paramount importance in the past few years. Indeed, it has been shown that state-of-the-art deep learning architectures can easily be fooled with imperceptible changes to their inputs. Therefore, finding good measures of robustness of a trained classifier is a key issue in the field. In this paper, we point out that averaging the radius of robustness of samples in a validation set is a statistically weak measure. We propose instead to weight the importance of samples depending on their difficulty. We motivate the proposed score by a theoretical case study using logistic regression, where we show that the proposed score is independent of the choice of the samples it is evaluated upon. We also empirically demonstrate the ability of the proposed score to measure robustness of classifiers with little dependence on the choice of samples in more complex settings, including deep convolutional neural networks and real datasets.
△ Less
Submitted 11 June, 2020; v1 submitted 9 June, 2020;
originally announced June 2020.
-
Improving Accuracy of Nonparametric Transfer Learning via Vector Segmentation
Authors:
Vincent Gripon,
Ghouthi B. Hacene,
Matthias Löwe,
Franck Vermet
Abstract:
Transfer learning using deep neural networks as feature extractors has become increasingly popular over the past few years. It allows to obtain state-of-the-art accuracy on datasets too small to train a deep neural network on its own, and it provides cutting edge descriptors that, combined with nonparametric learning methods, allow rapid and flexible deployment of performing solutions in computati…
▽ More
Transfer learning using deep neural networks as feature extractors has become increasingly popular over the past few years. It allows to obtain state-of-the-art accuracy on datasets too small to train a deep neural network on its own, and it provides cutting edge descriptors that, combined with nonparametric learning methods, allow rapid and flexible deployment of performing solutions in computationally restricted settings. In this paper, we are interested in showing that the features extracted using deep neural networks have specific properties which can be used to improve accuracy of downstream nonparametric learning methods. Namely, we demonstrate that for some distributions where information is embedded in a few coordinates, segmenting feature vectors can lead to better accuracy. We show how this model can be applied to real datasets by performing experiments using three mainstream deep neural network feature extractors and four databases, in vision and audio.
△ Less
Submitted 24 October, 2017;
originally announced October 2017.
-
Associative Memories to Accelerate Approximate Nearest Neighbor Search
Authors:
Vincent Gripon,
Matthias Löwe,
Franck Vermet
Abstract:
Nearest neighbor search is a very active field in machine learning for it appears in many application cases, including classification and object retrieval. In its canonical version, the complexity of the search is linear with both the dimension and the cardinal of the collection of vectors the search is performed in. Recently many works have focused on reducing the dimension of vectors using quant…
▽ More
Nearest neighbor search is a very active field in machine learning for it appears in many application cases, including classification and object retrieval. In its canonical version, the complexity of the search is linear with both the dimension and the cardinal of the collection of vectors the search is performed in. Recently many works have focused on reducing the dimension of vectors using quantization techniques or hashing, while providing an approximate result. In this paper we focus instead on tackling the cardinal of the collection of vectors. Namely, we introduce a technique that partitions the collection of vectors and stores each part in its own associative memory. When a query vector is given to the system, associative memories are polled to identify which one contain the closest match. Then an exhaustive search is conducted only on the part of vectors stored in the selected associative memory. We study the effectiveness of the system when messages to store are generated from i.i.d. uniform $\pm$1 random variables or 0-1 sparse i.i.d. random variables. We also conduct experiment on both synthetic data and real data and show it is possible to achieve interesting trade-offs between complexity and accuracy.
△ Less
Submitted 5 July, 2017; v1 submitted 10 November, 2016;
originally announced November 2016.
-
Characterizing Van Kampen Squares via Descent Data
Authors:
Harald König,
Uwe Wolter,
Michael Löwe
Abstract:
Categories in which cocones satisfy certain exactness conditions w.r.t. pullbacks are subject to current research activities in theoretical computer science. Usually, exactness is expressed in terms of properties of the pullback functor associated with the cocone. Even in the case of non-exactness, researchers in model semantics and rewriting theory inquire an elementary characterization of the im…
▽ More
Categories in which cocones satisfy certain exactness conditions w.r.t. pullbacks are subject to current research activities in theoretical computer science. Usually, exactness is expressed in terms of properties of the pullback functor associated with the cocone. Even in the case of non-exactness, researchers in model semantics and rewriting theory inquire an elementary characterization of the image of this functor. In this paper we will investigate this question in the special case where the cocone is a cospan, i.e. part of a Van Kampen square. The use of Descent Data as the dominant categorical tool yields two main results: A simple condition which characterizes the reachable part of the above mentioned functor in terms of liftings of involved equivalence relations and (as a consequence) a necessary and sufficient condition for a pushout to be a Van Kampen square formulated in a purely algebraic manner.
△ Less
Submitted 6 September, 2012;
originally announced September 2012.
