Showing 1–10 of 10 results for author: Vieille, N

Beyond discounted returns: Robust Markov decision processes with average and Blackwell optimality

Authors: Julien Grand-Clément, Marek Petrik, Nicolas Vieille

Abstract: Robust Markov Decision Processes (RMDPs) are a widely used framework for sequential decision-making under parameter uncertainty. RMDPs have been extensively studied when the objective is to maximize the discounted return, but little is known for average optimality (optimizing the long-run average of the rewards obtained over time) and Blackwell optimality (remaining discount optimal for all discou… ▽ More Robust Markov Decision Processes (RMDPs) are a widely used framework for sequential decision-making under parameter uncertainty. RMDPs have been extensively studied when the objective is to maximize the discounted return, but little is known for average optimality (optimizing the long-run average of the rewards obtained over time) and Blackwell optimality (remaining discount optimal for all discount factors sufficiently close to ). In this paper, we prove several foundational results for RMDPs beyond the discounted return. We show that average optimal policies can be chosen stationary and deterministic for sa-rectangular RMDPs but, perhaps surprisingly, we show that for s-rectangular RMDPs average optimal policies may not exist, and if they exist, may need to be history-dependent (Markovian). We also study Blackwell optimality for sa-rectangular RMDPs, where we show that $ε$-Blackwell optimal policies always exist, although Blackwell optimal policies may not exist. We also provide a sufficient condition for their existence, which encompasses virtually any examples from the literature. We then discuss the connection between average and Blackwell optimality, and we describe several algorithms to compute the optimal average return. Interestingly, our approach leverages the connections between RMDPs and stochastic games. Overall, our paper emphasizes the superior practical properties of distance-based sa-rectangular models over s-rectangular models for average and Blackwell optimality. △ Less

Submitted 14 January, 2025; v1 submitted 6 December, 2023; originally announced December 2023.

Optimal Dynamic Information Provision

Authors: Jerome Renault, Eilon Solan, Nicolas Vieille

Abstract: We study a dynamic model of information provision. A state of nature evolves according to a Markov chain. An informed advisor decides how much information to provide to an uninformed decision maker, so as to influence his short-term decisions. We deal with a stylized class of situations, in which the decision maker has a risky action and a safe action, and the payoff to the advisor only depends on… ▽ More We study a dynamic model of information provision. A state of nature evolves according to a Markov chain. An informed advisor decides how much information to provide to an uninformed decision maker, so as to influence his short-term decisions. We deal with a stylized class of situations, in which the decision maker has a risky action and a safe action, and the payoff to the advisor only depends on the action chosen by the decision maker. The greedy disclosure policy is the policy which, at each round, minimizes the amount of information being disclosed in that round, under the constraint that it maximizes the current payoff of the advisor. We prove that the greedy policy is optimal in many cases -- but not always. △ Less

Submitted 21 July, 2014; originally announced July 2014.

MSC Class: 91A10; 91A20; 90C40; 93C41

Markov games with frequent actions and incomplete information

Authors: Pierre Cardaliaguet, Catherine Rainer, Dinah Rosenberg, Nicolas Vieille

Abstract: We study a two-player, zero-sum, stochastic game with incomplete information on one side in which the players are allowed to play more and more frequently. The informed player observes the realization of a Markov chain on which the payoffs depend, while the non-informed player only observes his opponent's actions. We show the existence of a limit value as the time span between two consecutive stag… ▽ More We study a two-player, zero-sum, stochastic game with incomplete information on one side in which the players are allowed to play more and more frequently. The informed player observes the realization of a Markov chain on which the payoffs depend, while the non-informed player only observes his opponent's actions. We show the existence of a limit value as the time span between two consecutive stages vanishes; this value is characterized through an auxiliary optimization problem and as the solution of an Hamilton-Jacobi equation. △ Less

Submitted 12 July, 2013; originally announced July 2013.

Random Stopping Times in Stopping Problems and Stopping Games

Authors: Eilon Solan, Boris Tsirelson, Nicolas Vieille

Abstract: Three notions of random stopping times exist in the literature. We introduce two concepts of equivalence of random stopping times, motivated by optimal stopping problems and stopping games respectively. We prove that these two concepts coincide and that the three notions of random stopping times are equivalent. Three notions of random stopping times exist in the literature. We introduce two concepts of equivalence of random stopping times, motivated by optimal stopping problems and stopping games respectively. We prove that these two concepts coincide and that the three notions of random stopping times are equivalent. △ Less

Submitted 25 November, 2012; originally announced November 2012.

MSC Class: 60G40; 62L15

Dynamic Sender-Receiver Games

Authors: Jerome Renault, Eilon Solan, Nicolas Vieille

Abstract: We consider a dynamic version of sender-receiver games, where the sequence of states follows an irreducible Markov chain observed by the sender. Under mild assumptions, we provide a simple characterization of the limit set of equilibrium payoffs, as players become very patient. Under these assumptions, the limit set depends on the Markov chain only through its invariant measure. The (limit) equili… ▽ More We consider a dynamic version of sender-receiver games, where the sequence of states follows an irreducible Markov chain observed by the sender. Under mild assumptions, we provide a simple characterization of the limit set of equilibrium payoffs, as players become very patient. Under these assumptions, the limit set depends on the Markov chain only through its invariant measure. The (limit) equilibrium payoffs are the feasible payoffs that satisfy an individual rationality condition for the receiver, and an incentive compatibility condition for the sender. △ Less

Submitted 2 April, 2012; originally announced April 2012.

MSC Class: 60J10; 91A05; 91A10; 91A20

Strategic Information Exchange

Authors: Dinah Rosenberg, Eilon Solan, Nicolas Vieille

Abstract: We study a class of two-player repeated games with incomplete information and informational externalities. In these games, two states are chosen at the outset, and players get private information on the pair, before engaging in repeated play. The payoff of each player only depends on his `own' state and on his own action. We study to what extent, and how, information can be exchanged in equilibriu… ▽ More We study a class of two-player repeated games with incomplete information and informational externalities. In these games, two states are chosen at the outset, and players get private information on the pair, before engaging in repeated play. The payoff of each player only depends on his `own' state and on his own action. We study to what extent, and how, information can be exchanged in equilibrium. We prove that provided the private information of each player is valuable for the other player, the set of sequential equilibrium payoffs converges to the set of feasible and individually rational payoffs as players become patient. △ Less

Submitted 26 July, 2010; originally announced July 2010.

Lowest Unique Bid Auctions

Authors: Marco Scarsini, Eilon Solan, Nicolas Vieille

Abstract: We consider a class of auctions (Lowest Unique Bid Auctions) that have achieved a considerable success on the Internet. Bids are made in cents (of euro) and every bidder can bid as many numbers as she wants. The lowest unique bid wins the auction. Every bid has a fixed cost, and once a participant makes a bid, she gets to know whether her bid was unique and whether it was the lowest unique. Inform… ▽ More We consider a class of auctions (Lowest Unique Bid Auctions) that have achieved a considerable success on the Internet. Bids are made in cents (of euro) and every bidder can bid as many numbers as she wants. The lowest unique bid wins the auction. Every bid has a fixed cost, and once a participant makes a bid, she gets to know whether her bid was unique and whether it was the lowest unique. Information is updated in real time, but every bidder sees only what's relevant to the bids she made. We show that the observed behavior in these auctions differs considerably from what theory would prescribe if all bidders were fully rational. We show that the seller makes money, which would not be the case with rational bidders, and some bidders win the auctions quite often. We describe a possible strategy for these bidders. △ Less

Submitted 24 July, 2010; originally announced July 2010.

On the Optimal Amount of Experimentation in Sequential Decision Problems

Authors: Dinah Rosenberg, Eilon Solan, Nicolas Vieille

Abstract: We provide a tight bound on the amount of experimentation under the optimal strategy in sequential decision problems. We show the applicability of the result by providing a bound on the cut-off in a one-arm bandit problem. We provide a tight bound on the amount of experimentation under the optimal strategy in sequential decision problems. We show the applicability of the result by providing a bound on the cut-off in a one-arm bandit problem. △ Less

Submitted 12 July, 2009; originally announced July 2009.

MSC Class: 62C10; 60G99; 93E35

Approximating a sequence of observations by a simple process

Authors: Dinah Rosenberg, Eilon Solan, Nicolas Vieille

Abstract: Given an arbitrary long but finite sequence of observations from a finite set, we construct a simple process that approximates the sequence, in the sense that with high probability the empirical frequency, as well as the empirical one-step transitions along a realization from the approximating process, are close to that of the given sequence. We generalize the result to the case where the one-st… ▽ More Given an arbitrary long but finite sequence of observations from a finite set, we construct a simple process that approximates the sequence, in the sense that with high probability the empirical frequency, as well as the empirical one-step transitions along a realization from the approximating process, are close to that of the given sequence. We generalize the result to the case where the one-step transitions are required to be in given polyhedra. △ Less

Submitted 30 August, 2005; originally announced August 2005.

Comments: Published at http://dx.doi.org/10.1214/009053604000000643 in the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-AOS-AOS222 MSC Class: 60J99; 62M09; 93E03. (Primary)

Journal ref: Annals of Statistics 2004, Vol. 32, No. 6, 2742-2775

Perturbed Markov Chains

Authors: Eilon Solan, Nicolas Vieille

Abstract: We study irreducible time-homogenous Markov chains with finite state space in discrete time. We obtain results on the sensitivity of the stationary distribution and other statistical quantities with respect to perturbations of the transition matrix. We define a new closeness relation between transition matrices, and use graph-theoretic techniques, in contrast with the matrix analysis techniques… ▽ More We study irreducible time-homogenous Markov chains with finite state space in discrete time. We obtain results on the sensitivity of the stationary distribution and other statistical quantities with respect to perturbations of the transition matrix. We define a new closeness relation between transition matrices, and use graph-theoretic techniques, in contrast with the matrix analysis techniques previously used. △ Less

Submitted 17 June, 2003; originally announced June 2003.

Comments: 22 pages

MSC Class: 60J10; 60F10

Journal ref: Journal of Applied Probability, 2003, 40, 107-122