-
On Tilted Losses in Machine Learning: Theory and Applications
Authors:
Tian Li,
Ahmad Beirami,
Maziar Sanjabi,
Virginia Smith
Abstract:
Exponential tilting is a technique commonly used in fields such as statistics, probability, information theory, and optimization to create parametric distribution shifts. Despite its prevalence in related fields, tilting has not seen widespread use in machine learning. In this work, we aim to bridge this gap by exploring the use of tilting in risk minimization. We study a simple extension to ERM -…
▽ More
Exponential tilting is a technique commonly used in fields such as statistics, probability, information theory, and optimization to create parametric distribution shifts. Despite its prevalence in related fields, tilting has not seen widespread use in machine learning. In this work, we aim to bridge this gap by exploring the use of tilting in risk minimization. We study a simple extension to ERM -- tilted empirical risk minimization (TERM) -- which uses exponential tilting to flexibly tune the impact of individual losses. The resulting framework has several useful properties: We show that TERM can increase or decrease the influence of outliers, respectively, to enable fairness or robustness; has variance-reduction properties that can benefit generalization; and can be viewed as a smooth approximation to the tail probability of losses. Our work makes rigorous connections between TERM and related objectives, such as Value-at-Risk, Conditional Value-at-Risk, and distributionally robust optimization (DRO). We develop batch and stochastic first-order optimization methods for solving TERM, provide convergence guarantees for the solvers, and show that the framework can be efficiently solved relative to common alternatives. Finally, we demonstrate that TERM can be used for a multitude of applications in machine learning, such as enforcing fairness between subgroups, mitigating the effect of outliers, and handling class imbalance. Despite the straightforward modification TERM makes to traditional ERM objectives, we find that the framework can consistently outperform ERM and deliver competitive performance with state-of-the-art, problem-specific approaches.
△ Less
Submitted 1 June, 2023; v1 submitted 13 September, 2021;
originally announced September 2021.
-
Global Games with Noisy Information Sharing
Authors:
Hessam Mahdavifar,
Ahmad Beirami,
Behrouz Touri,
Jeff S. Shamma
Abstract:
Global games form a subclass of games with incomplete information where a set of agents decide actions against a regime with an underlying fundamental $θ$ representing its power. Each agent has access to an independent noisy observation of $θ$. In order to capture the behavior of agents in a social network of information exchange we assume that agents share their observation in a noisy environment…
▽ More
Global games form a subclass of games with incomplete information where a set of agents decide actions against a regime with an underlying fundamental $θ$ representing its power. Each agent has access to an independent noisy observation of $θ$. In order to capture the behavior of agents in a social network of information exchange we assume that agents share their observation in a noisy environment prior to making their decision. We show that global games with noisy sharing of information do not admit an intuitive type of threshold policy which only depends on agents' belief about the underlying $θ$. This is in contrast to the existing results on the threshold policy for the conventional set-up of global games. Motivated by this result, we investigate the existence of equilibrium strategies in a more general collection of threshold-type policies and show that such equilibrium strategies exist and are unique if the sharing of information happens over a sufficiently noisy environment.
△ Less
Submitted 28 October, 2017; v1 submitted 28 October, 2015;
originally announced October 2015.
-
A Framework for Investigating the Performance of Chaotic-Map Truly Random Number Generators
Authors:
Ahmad Beirami,
Hamid Nejati
Abstract:
In this paper, we approximate the hidden Markov model of chaotic-map truly random number generators (TRNGs) and describe its fundamental limits based on the approximate entropy-rate of the underlying bit-generation process. We demonstrate that entropy-rate plays a key role in the performance and robustness of chaotic-map TRNGs, which must be taken into account in the circuit design optimization. W…
▽ More
In this paper, we approximate the hidden Markov model of chaotic-map truly random number generators (TRNGs) and describe its fundamental limits based on the approximate entropy-rate of the underlying bit-generation process. We demonstrate that entropy-rate plays a key role in the performance and robustness of chaotic-map TRNGs, which must be taken into account in the circuit design optimization. We further derive optimality conditions for post-processing units that extract truly random bits from a raw-RNG.
△ Less
Submitted 1 March, 2013; v1 submitted 6 November, 2012;
originally announced November 2012.
-
Discrete-Time Chaotic-Map Truly Random Number Generators: Design, Implementation, and Variability Analysis of the Zigzag Map
Authors:
Hamid Nejati,
Ahmad Beirami,
Warsame H. Ali
Abstract:
In this paper, we introduce a novel discrete chaotic map named zigzag map that demonstrates excellent chaotic behaviors and can be utilized in Truly Random Number Generators (TRNGs). We comprehensively investigate the map and explore its critical chaotic characteristics and parameters. We further present two circuit implementations for the zigzag map based on the switched current technique as well…
▽ More
In this paper, we introduce a novel discrete chaotic map named zigzag map that demonstrates excellent chaotic behaviors and can be utilized in Truly Random Number Generators (TRNGs). We comprehensively investigate the map and explore its critical chaotic characteristics and parameters. We further present two circuit implementations for the zigzag map based on the switched current technique as well as the current-mode affine interpolation of the breakpoints. In practice, implementation variations can deteriorate the quality of the output sequence as a result of variation of the chaotic map parameters. In order to quantify the impact of variations on the map performance, we model the variations using a combination of theoretical analysis and Monte-Carlo simulations on the circuits. We demonstrate that even in the presence of the map variations, a TRNG based on the zigzag map passes all of the NIST 800-22 statistical randomness tests using simple post processing of the output data.
△ Less
Submitted 5 June, 2012;
originally announced June 2012.
-
A Realizable Modified Tent Map for True Random Number Generation
Authors:
Hamid Nejati,
Ahmad Beirami,
Yehia Massoud
Abstract:
Tent map is a discrete-time piecewise-affine I/O characteristic curve, which is used for chaos-based applications, such as true random number generation. However, tent map suffers from the inability to maintain the output state confined to the input range under noise and process variations. In this paper, we propose a modified tent map, which is interchangeable with the tent map for practical appl…
▽ More
Tent map is a discrete-time piecewise-affine I/O characteristic curve, which is used for chaos-based applications, such as true random number generation. However, tent map suffers from the inability to maintain the output state confined to the input range under noise and process variations. In this paper, we propose a modified tent map, which is interchangeable with the tent map for practical applications. In the proposed modified tent map, the confinement problem is solved while maintaining the functionality of the tent map. We also demonstrate simulation results for the circuit implementation of the presented modified tent map for true random number generation.
△ Less
Submitted 22 May, 2012; v1 submitted 18 May, 2012;
originally announced May 2012.
-
A Performance Metric for Discrete-Time Chaos-Based Truly Random Number Generators
Authors:
Ahmad Beirami,
Hamid Nejati,
Yehia Massoud
Abstract:
In this paper, we develop an information entropy based metric that represents the statistical quality of the generated binary sequence in Truly Random Number Generators (TRNG). The metric can be used for the design and optimization of the TRNG circuits as well as the development of efficient post-processing units for recovering the degraded statistical characteristics of the signal due to process…
▽ More
In this paper, we develop an information entropy based metric that represents the statistical quality of the generated binary sequence in Truly Random Number Generators (TRNG). The metric can be used for the design and optimization of the TRNG circuits as well as the development of efficient post-processing units for recovering the degraded statistical characteristics of the signal due to process variations.
△ Less
Submitted 10 January, 2012;
originally announced January 2012.
