Showing 1–21 of 21 results for author: Manski, C F

A Decision Theoretic Perspective on Artificial Superintelligence: Coping with Missing Data Problems in Prediction and Treatment Choice

Authors: Jeff Dominitz, Charles F. Manski

Abstract: Enormous attention and resources are being devoted to the quest for artificial general intelligence and, even more ambitiously, artificial superintelligence. We wonder about the implications for our methodological research, which aims to help decision makers cope with what econometricians call identification problems, inferential problems in empirical research that do not diminish as sample size g… ▽ More Enormous attention and resources are being devoted to the quest for artificial general intelligence and, even more ambitiously, artificial superintelligence. We wonder about the implications for our methodological research, which aims to help decision makers cope with what econometricians call identification problems, inferential problems in empirical research that do not diminish as sample size grows. Of particular concern are missing data problems in prediction and treatment choice. Essentially all data collection intended to inform decision making is subject to missing data, which gives rise to identification problems. Thus far, we see no indication that the current dominant architecture of machine learning (ML)-based artificial intelligence (AI) systems will outperform humans in this context. In this paper, we explain why we have reached this conclusion and why we see the missing data problem as a cautionary case study in the quest for superintelligence more generally. We first discuss the concept of intelligence, before presenting a decision-theoretic perspective that formalizes the connection between intelligence and identification problems. We next apply this perspective to two leading cases of missing data problems. Then we explain why we are skeptical that AI research is currently on a path toward machines doing better than humans at solving these identification problems. △ Less

Submitted 15 September, 2025; originally announced September 2025.

Utilitarian or Quantile-Welfare Evaluation of Health Policy?

Authors: Charles F. Manski, John Mullahy

Abstract: This paper considers quantile-welfare evaluation of health policy as an alternative to utilitarian evaluation. Manski (1988) originally proposed and studied maximization of quantile utility as a model of individual decision making under uncertainty, juxtaposing it with maximization of expected utility. That paper's primary motivation was to exploit the fact that maximization of quantile utility re… ▽ More This paper considers quantile-welfare evaluation of health policy as an alternative to utilitarian evaluation. Manski (1988) originally proposed and studied maximization of quantile utility as a model of individual decision making under uncertainty, juxtaposing it with maximization of expected utility. That paper's primary motivation was to exploit the fact that maximization of quantile utility requires only an ordinal formalization of utility, not a cardinal one. This paper transfers these ideas from analysis of individual decision making to analysis of social planning. We begin by summarizing basic theoretical properties of quantile welfare in general terms rather than related specifically to health policy. We then propose a procedure to nonparametrically bound the quantile welfare of health states using data from binary-choice time-tradeoff (TTO) experiments of the type regularly performed by health economists. After this we assess related econometric considerations concerning measurement, using the EQ-5D framework to structure our discussion. △ Less

Submitted 5 September, 2025; originally announced September 2025.

What is the general Welfare? Welfare Economic Perspectives

Authors: Charles F. Manski

Abstract: Researchers do not know what the framers of the United States Constitution intended when they wrote of the general Welfare. Nevertheless, economists can conjecture by specifying social welfare functions that aim to express the preferences of the population. Economists have often simplified utilitarian analysis of public policy by assuming that individuals have homogeneous, consequentialist, and se… ▽ More Researchers do not know what the framers of the United States Constitution intended when they wrote of the general Welfare. Nevertheless, economists can conjecture by specifying social welfare functions that aim to express the preferences of the population. Economists have often simplified utilitarian analysis of public policy by assuming that individuals have homogeneous, consequentialist, and self-centered preferences. In reality, individuals may hold heterogeneous private and social preferences. To enhance policy analysis, I argue that economists should specify social welfare functions that express the richness and variety of actual personal preferences over social states. The possibilities are vast. I focus on preferences for population prosperity and equity. There has been much controversy regarding interpretation of equity, a term that public discourse has used in vague and conflicting ways. Specifying social welfare functions that formally express different interpretations of equity may not eliminate disagreements, but it should clarify concepts of equity and reduce the inconsistencies that afflict verbal communication. △ Less

Submitted 26 June, 2025; v1 submitted 14 January, 2025; originally announced January 2025.

Prediction with Differential Covariate Classification: Illustrated by Racial/Ethnic Classification in Medical Risk Assessment

Authors: Charles F. Manski, John Mullahy, Atheendar S. Venkataramani

Abstract: A common practice in evidence-based decision-making uses estimates of conditional probabilities P(y|x) obtained from research studies to predict outcomes y on the basis of observed covariates x. Given this information, decisions are then based on the predicted outcomes. Researchers commonly assume that the predictors used in the generation of the evidence are the same as those used in applying the… ▽ More A common practice in evidence-based decision-making uses estimates of conditional probabilities P(y|x) obtained from research studies to predict outcomes y on the basis of observed covariates x. Given this information, decisions are then based on the predicted outcomes. Researchers commonly assume that the predictors used in the generation of the evidence are the same as those used in applying the evidence: i.e., the meaning of x in the two circumstances is the same. This may not be the case in real-world settings. Across a wide-range of settings, ranging from clinical practice or education policy, demographic attributes (e.g., age, race, ethnicity) are often classified differently in research studies than in decision settings. This paper studies identification in such settings. We propose a formal framework for prediction with what we term differential covariate classification (DCC). Using this framework, we analyze partial identification of probabilistic predictions and assess how various assumptions influence the identification regions. We apply the findings to a range of settings, focusing mainly on differential classification of individuals' race and ethnicity in clinical medicine. We find that bounds on P(y|x) can be wide, and the information needed to narrow them available only in special cases. These findings highlight an important problem in using evidence in decision making, a problem that has not yet been fully appreciated in debates on classification in public policy and medicine. △ Less

Submitted 4 January, 2025; originally announced January 2025.

Using Ordinal Voting to Compare the Utilitarian Welfare of a Status Quo and A Proposed Policy: A Simple Nonparametric Analysis

Authors: Charles F. Manski

Abstract: The relationship of policy choice by majority voting and by maximization of utilitarian welfare has long been discussed. I consider choice between a status quo and a proposed policy when persons have interpersonally comparable cardinal utilities taking values in a bounded interval, voting is compulsory, and each person votes for a policy that maximizes utility. I show that knowledge of the attaine… ▽ More The relationship of policy choice by majority voting and by maximization of utilitarian welfare has long been discussed. I consider choice between a status quo and a proposed policy when persons have interpersonally comparable cardinal utilities taking values in a bounded interval, voting is compulsory, and each person votes for a policy that maximizes utility. I show that knowledge of the attained status quo welfare and the voting outcome yields an informative bound on welfare with the proposed policy. The bound contains the value of status quo welfare, so the better utilitarian policy is not known. The minimax-regret decision and certain Bayes decisions choose the proposed policy if its vote share exceeds the known value of status quo welfare. This procedure differs from majority rule, which chooses the proposed policy if its vote share exceeds 1/2. △ Less

Submitted 24 December, 2024; originally announced December 2024.

The Subtlety of Optimal Paternalism in a Population with Bounded Rationality

Authors: Charles F. Manski, Eytan Sheshinski

Abstract: We study optimal policy when a paternalistic utilitarian planner has the power to design a discrete choice set for a heterogeneous population with bounded rationality. We show that the policy that most effectively constrains or influences choices depends in a particular multiplicative way on the preferences of the population and on the choice probabilities conditional on preferences that measure t… ▽ More We study optimal policy when a paternalistic utilitarian planner has the power to design a discrete choice set for a heterogeneous population with bounded rationality. We show that the policy that most effectively constrains or influences choices depends in a particular multiplicative way on the preferences of the population and on the choice probabilities conditional on preferences that measure the suboptimality of behavior. We first consider the planning problem in abstraction. We then study two settings in which the planner may mandate an action or decentralize decision making. In one setting, we suppose that individuals measure utility with additive random error and maximize mismeasured rather than actual utility. Then optimal planning requires knowledge of the distribution of measurement errors. In the second setting, we consider binary treatment choice under uncertainty when the planner can mandate a treatment conditional on publicly observed personal covariates or can enable individuals to choose their own treatments conditional on private information. We focus on situations where bounded rationality takes the form of deviations between subjective personal beliefs and objective probabilities of uncertain outcomes. To illustrate, we consider clinical decision making in medicine. In toto, our analysis is cautionary. It characterizes the subtle nature of optimal policy, whose determination requires the planner to possess extensive knowledge that is rarely available. We conclude that studies of policy choice by a paternalistic utilitarian planner should view not only the population but also the planner to be boundedly rational. △ Less

Submitted 15 August, 2025; v1 submitted 17 October, 2024; originally announced October 2024.

Using Total Margin of Error to Account for Non-Sampling Error in Election Polls: The Case of Nonresponse

Authors: Jeff Dominitz, Charles F. Manski

Abstract: The potential impact of non-sampling errors on election polls is well known, but measurement has focused on the margin of sampling error. Survey statisticians have long recommended measurement of total survey error by mean square error (MSE), which jointly measures sampling and non-sampling errors. We think it reasonable to use the square root of maximum MSE to measure the total margin of error (T… ▽ More The potential impact of non-sampling errors on election polls is well known, but measurement has focused on the margin of sampling error. Survey statisticians have long recommended measurement of total survey error by mean square error (MSE), which jointly measures sampling and non-sampling errors. We think it reasonable to use the square root of maximum MSE to measure the total margin of error (TME). Measurement of TME should encompass both sampling error and all forms of non-sampling error. We suggest that measurement of TME should be a standard feature in the reporting of polls. To provide a clear illustration, and because we believe the exceedingly low response rates commonly obtained by election polls to be a particularly worrisome source of potential error, we demonstrate how to measure the potential impact of nonresponse using the concept of TME. We first show how to measure TME when a pollster lacks any knowledge of the candidate preferences of nonrespondents. We then extend the analysis to settings where the pollster has partial knowledge that bounds the preferences of non-respondents. In each setting, we derive a simple poll estimate that approximately minimizes TME, a midpoint estimate, and compare it to a conventional poll estimate. △ Less

Submitted 31 October, 2024; v1 submitted 27 July, 2024; originally announced July 2024.

Comprehensive OOS Evaluation of Predictive Algorithms with Statistical Decision Theory

Authors: Jeff Dominitz, Charles F. Manski

Abstract: We argue that comprehensive out-of-sample (OOS) evaluation using statistical decision theory (SDT) should replace the current practice of K-fold and Common Task Framework validation in machine learning (ML) research on prediction. SDT provides a formal frequentist framework for performing comprehensive OOS evaluation across all possible (1) training samples, (2) populations that may generate train… ▽ More We argue that comprehensive out-of-sample (OOS) evaluation using statistical decision theory (SDT) should replace the current practice of K-fold and Common Task Framework validation in machine learning (ML) research on prediction. SDT provides a formal frequentist framework for performing comprehensive OOS evaluation across all possible (1) training samples, (2) populations that may generate training data, and (3) populations of prediction interest. Regarding feature (3), we emphasize that SDT requires the practitioner to directly confront the possibility that the future may not look like the past and to account for a possible need to extrapolate from one population to another when building a predictive algorithm. For specificity, we consider treatment choice using conditional predictions with alternative restrictions on the state space of possible populations that may generate training data. We discuss application of SDT to the problem of predicting patient illness to inform clinical decision making. SDT is simple in abstraction, but it is often computationally demanding to implement. We call on ML researchers, econometricians, and statisticians to expand the domain within which implementation of SDT is tractable. △ Less

Submitted 16 April, 2025; v1 submitted 16 March, 2024; originally announced March 2024.

Comments: arXiv admin note: text overlap with arXiv:2110.00864

Statistical Decision Theory Respecting Stochastic Dominance

Authors: Charles F. Manski, Aleksey Tetenov

Abstract: The statistical decision theory pioneered by Wald (1950) has used state-dependent mean loss (risk) to measure the performance of statistical decision functions across potential samples. We think it evident that evaluation of performance should respect stochastic dominance, but we do not see a compelling reason to focus exclusively on mean loss. We think it instructive to also measure performance b… ▽ More The statistical decision theory pioneered by Wald (1950) has used state-dependent mean loss (risk) to measure the performance of statistical decision functions across potential samples. We think it evident that evaluation of performance should respect stochastic dominance, but we do not see a compelling reason to focus exclusively on mean loss. We think it instructive to also measure performance by other functionals that respect stochastic dominance, such as quantiles of the distribution of loss. This paper develops general principles and illustrative applications for statistical decision theory respecting stochastic dominance. We modify the Wald definition of admissibility to an analogous concept of stochastic dominance (SD) admissibility, which uses stochastic dominance rather than mean sampling performance to compare alternative decision rules. We study SD admissibility in two relatively simple classes of decision problems that arise in treatment choice. We reevaluate the relationship between the MLE, James-Stein, and James-Stein positive part estimators from the perspective of SD admissibility. We consider alternative criteria for choice among SD-admissible rules. We juxtapose traditional criteria based on risk, regret, or Bayes risk with analogous ones based on quantiles of state-dependent sampling distributions or the Bayes distribution of loss. △ Less

Submitted 9 August, 2023; originally announced August 2023.

Using Limited Trial Evidence to Credibly Choose Treatment Dosage when Efficacy and Adverse Effects Weakly Increase with Dose

Authors: Charles F. Manski

Abstract: In medical treatment and elsewhere, it has become standard to base treatment intensity (dosage) on evidence in randomized trials. Yet it has been rare to study how outcomes vary with dosage. In trials to obtain drug approval, the norm has been to specify some dose of a new drug and compare it with an established therapy or placebo. Design-based trial analysis views each trial arm as qualitatively… ▽ More In medical treatment and elsewhere, it has become standard to base treatment intensity (dosage) on evidence in randomized trials. Yet it has been rare to study how outcomes vary with dosage. In trials to obtain drug approval, the norm has been to specify some dose of a new drug and compare it with an established therapy or placebo. Design-based trial analysis views each trial arm as qualitatively different, but it may be highly credible to assume that efficacy and adverse effects (AEs) weakly increase with dosage. Optimization of patient care requires joint attention to both, as well as to treatment cost. This paper develops methodology to credibly use limited trial evidence to choose dosage when efficacy and AEs weakly increase with dose. I suppose that dosage is an integer choice t in (0, 1, . . . , T), T being a specified maximum dose. I study dosage choice when trial evidence on outcomes is available for only K dose levels, where K < T + 1. Then the population distribution of dose response is partially rather than point identified. The identification region is a convex polygon determined by linear equalities and inequalities. I characterize clinical and public-health decision making using the minimax-regret criterion. A simple analytical solution exists when T = 2 and computation is tractable when T is larger. △ Less

Submitted 26 May, 2023; originally announced May 2023.

Partial Identification of Personalized Treatment Response with Trial-reported Analyses of Binary Subgroups

Authors: Sheyu Li, Valentyn Litvin, Charles F. Manski

Abstract: Medical journals have adhered to a reporting practice that seriously limits the usefulness of published trial findings. Medical decision makers commonly observe many patient covariates and seek to use this information to personalize treatment choices. Yet standard summaries of trial findings only partition subjects into broad subgroups, typically into binary categories. Given this reporting practi… ▽ More Medical journals have adhered to a reporting practice that seriously limits the usefulness of published trial findings. Medical decision makers commonly observe many patient covariates and seek to use this information to personalize treatment choices. Yet standard summaries of trial findings only partition subjects into broad subgroups, typically into binary categories. Given this reporting practice, we study the problem of inference on long mean treatment outcomes E[y(t)|x], where t is a treatment, y(t) is a treatment outcome, and the covariate vector x has length K, each component being a binary variable. The available data are estimates of {E[y(t)|xk = 0], E[y(t)|xk = 1], P(xk)}, k = 1, . . . , K reported in journal articles. We show that reported trial findings partially identify {E[y(t)|x], P(x)}. Illustrative computations demonstrate that the summaries of trial findings in journal articles may imply only wide bounds on long mean outcomes. One can realistically tighten inferences if one can combine reported trial findings with credible assumptions having identifying power, such as bounded-variation assumptions. △ Less

Submitted 3 September, 2022; v1 submitted 5 August, 2022; originally announced August 2022.

Inference with Imputed Data: The Allure of Making Stuff Up

Authors: Charles F. Manski

Abstract: Incomplete observability of data generates an identification problem. There is no panacea for missing data. What one can learn about a population parameter depends on the assumptions one finds credible to maintain. The credibility of assumptions varies with the empirical setting. No specific assumptions can provide a realistic general solution to the problem of inference with missing data. Yet Rub… ▽ More Incomplete observability of data generates an identification problem. There is no panacea for missing data. What one can learn about a population parameter depends on the assumptions one finds credible to maintain. The credibility of assumptions varies with the empirical setting. No specific assumptions can provide a realistic general solution to the problem of inference with missing data. Yet Rubin has promoted random multiple imputation (RMI) as a general way to deal with missing values in public-use data. This recommendation has been influential to empirical researchers who seek a simple fix to the nuisance of missing data. This paper adds to my earlier critiques of imputation. It provides a transparent assessment of the mix of Bayesian and frequentist thinking used by Rubin to argue for RMI. It evaluates random imputation to replace missing outcome or covariate data when the objective is to learn a conditional expectation. It considers steps that might help combat the allure of making stuff up. △ Less

Submitted 15 May, 2022; originally announced May 2022.

Comments: arXiv admin note: text overlap with arXiv:2102.11334

Identification and Statistical Decision Theory

Authors: Charles F. Manski

Abstract: Econometricians have usefully separated study of estimation into identification and statistical components. Identification analysis, which assumes knowledge of the probability distribution generating observable data, places an upper bound on what may be learned about population parameters of interest with finite sample data. Yet Wald's statistical decision theory studies decision making with sampl… ▽ More Econometricians have usefully separated study of estimation into identification and statistical components. Identification analysis, which assumes knowledge of the probability distribution generating observable data, places an upper bound on what may be learned about population parameters of interest with finite sample data. Yet Wald's statistical decision theory studies decision making with sample data without reference to identification, indeed without reference to estimation. This paper asks if identification analysis is useful to statistical decision theory. The answer is positive, as it can yield an informative and tractable upper bound on the achievable finite sample performance of decision criteria. The reasoning is simple when the decision relevant parameter is point identified. It is more delicate when the true state is partially identified and a decision must be made under ambiguity. Then the performance of some criteria, such as minimax regret, is enhanced by randomizing choice of an action. This may be accomplished by making choice a function of sample data. I find it useful to recast choice of a statistical decision function as selection of choice probabilities for the elements of the choice set. Using sample data to randomize choice conceptually differs from and is complementary to its traditional use to estimate population parameters. △ Less

Submitted 21 March, 2024; v1 submitted 24 April, 2022; originally announced April 2022.

Journal ref: Econom. Theory 41 (2025) 977-993

Minimax-Regret Climate Policy with Deep Uncertainty in Climate Modeling and Intergenerational Discounting

Authors: Stephen J. DeCanio, Charles F. Manski, Alan H. Sanstad

Abstract: Integrated assessment models have become the primary tools for comparing climate policies that seek to reduce greenhouse gas emissions. Policy comparisons have often been performed by considering a planner who seeks to make optimal trade-offs between the costs of carbon abatement and the economic damages from climate change. The planning problem has been formalized as one of optimal control, the o… ▽ More Integrated assessment models have become the primary tools for comparing climate policies that seek to reduce greenhouse gas emissions. Policy comparisons have often been performed by considering a planner who seeks to make optimal trade-offs between the costs of carbon abatement and the economic damages from climate change. The planning problem has been formalized as one of optimal control, the objective being to minimize the total costs of abatement and damages over a time horizon. Studying climate policy as a control problem presumes that a planner knows enough to make optimization feasible, but physical and economic uncertainties abound. Earlier, Manski, Sanstad, and DeCanio proposed and studied use of the minimax-regret (MMR) decision criterion to account for deep uncertainty in climate modeling. Here we study choice of climate policy that minimizes maximum regret with deep uncertainty regarding both the correct climate model and the appropriate time discount rate to use in intergenerational assessment of policy consequences. The analysis specifies a range of discount rates to express both empirical and normative uncertainty about the appropriate rate. The findings regarding climate policy are novel and informative. The MMR analysis points to use of a relatively low discount rate of 0.02 for climate policy. The MMR decision rule keeps the maximum future temperature increase below 2C above the 1900-10 level for most of the parameter values used to weight costs and damages. △ Less

Submitted 21 January, 2022; originally announced January 2022.

Patient-Centered Appraisal of Race-Free Clinical Risk Assessment

Authors: Charles F. Manski

Abstract: Until recently, there has been a consensus that clinicians should condition patient risk assessments on all observed patient covariates with predictive power. The broad idea is that knowing more about patients enables more accurate predictions of their health risks and, hence, better clinical decisions. This consensus has recently unraveled with respect to a specific covariate, namely race. There… ▽ More Until recently, there has been a consensus that clinicians should condition patient risk assessments on all observed patient covariates with predictive power. The broad idea is that knowing more about patients enables more accurate predictions of their health risks and, hence, better clinical decisions. This consensus has recently unraveled with respect to a specific covariate, namely race. There have been increasing calls for race-free risk assessment, arguing that using race to predict patient outcomes contributes to racial disparities and inequities in health care. Writers calling for race-free risk assessment have not studied how it would affect the quality of clinical decisions. Considering the matter from the patient-centered perspective of medical economics yields a disturbing conclusion: Race-free risk assessment would harm patients of all races. △ Less

Submitted 26 February, 2022; v1 submitted 2 December, 2021; originally announced December 2021.

Probabilistic Prediction for Binary Treatment Choice: with focus on personalized medicine

Authors: Charles F. Manski

Abstract: This paper extends my research applying statistical decision theory to treatment choice with sample data, using maximum regret to evaluate the performance of treatment rules. The specific new contribution is to study as-if optimization using estimates of illness probabilities in clinical choice between surveillance and aggressive treatment. Beyond its specifics, the paper sends a broad message. St… ▽ More This paper extends my research applying statistical decision theory to treatment choice with sample data, using maximum regret to evaluate the performance of treatment rules. The specific new contribution is to study as-if optimization using estimates of illness probabilities in clinical choice between surveillance and aggressive treatment. Beyond its specifics, the paper sends a broad message. Statisticians and computer scientists have addressed conditional prediction for decision making in indirect ways, the former applying classical statistical theory and the latter measuring prediction accuracy in test samples. Neither approach is satisfactory. Statistical decision theory provides a coherent, generally applicable methodology. △ Less

Submitted 2 October, 2021; originally announced October 2021.

Misguided Use of Observed Covariates to Impute Missing Covariates in Conditional Prediction: A Shrinkage Problem

Authors: Charles F Manski, Michael Gmeiner, Anat Tamburc

Abstract: Researchers regularly perform conditional prediction using imputed values of missing data. However, applications of imputation often lack a firm foundation in statistical theory. This paper originated when we were unable to find analysis substantiating claims that imputation of missing data has good frequentist properties when data are missing at random (MAR). We focused on the use of observed cov… ▽ More Researchers regularly perform conditional prediction using imputed values of missing data. However, applications of imputation often lack a firm foundation in statistical theory. This paper originated when we were unable to find analysis substantiating claims that imputation of missing data has good frequentist properties when data are missing at random (MAR). We focused on the use of observed covariates to impute missing covariates when estimating conditional means of the form E(y|x, w). Here y is an outcome whose realizations are always observed, x is a covariate whose realizations are always observed, and w is a covariate whose realizations are sometimes unobserved. We examine the probability limit of simple imputation estimates of E(y|x, w) as sample size goes to infinity. We find that these estimates are not consistent when covariate data are MAR. To the contrary, the estimates suffer from a shrinkage problem. They converge to points intermediate between the conditional mean of interest, E(y|x, w), and the mean E(y|x) that conditions only on x. We use a type of genotype imputation to illustrate. △ Less

Submitted 22 February, 2021; originally announced February 2021.

Statistical Decision Properties of Imprecise Trials Assessing COVID-19 Drugs

Authors: Charles F. Manski, Aleksey Tetenov

Abstract: As the COVID-19 pandemic progresses, researchers are reporting findings of randomized trials comparing standard care with care augmented by experimental drugs. The trials have small sample sizes, so estimates of treatment effects are imprecise. Seeing imprecision, clinicians reading research articles may find it difficult to decide when to treat patients with experimental drugs. Whatever decision… ▽ More As the COVID-19 pandemic progresses, researchers are reporting findings of randomized trials comparing standard care with care augmented by experimental drugs. The trials have small sample sizes, so estimates of treatment effects are imprecise. Seeing imprecision, clinicians reading research articles may find it difficult to decide when to treat patients with experimental drugs. Whatever decision criterion one uses, there is always some probability that random variation in trial outcomes will lead to prescribing sub-optimal treatments. A conventional practice when comparing standard care and an innovation is to choose the innovation only if the estimated treatment effect is positive and statistically significant. This practice defers to standard care as the status quo. To evaluate decision criteria, we use the concept of near-optimality, which jointly considers the probability and magnitude of decision errors. An appealing decision criterion from this perspective is the empirical success rule, which chooses the treatment with the highest observed average patient outcome in the trial. Considering the design of recent and ongoing COVID-19 trials, we show that the empirical success rule yields treatment results that are much closer to optimal than those generated by prevailing decision criteria based on hypothesis tests. △ Less

Submitted 30 May, 2020; originally announced June 2020.

Comments: 28 pages, 4 tables

Estimating the COVID-19 Infection Rate: Anatomy of an Inference Problem

Authors: Charles F. Manski, Francesca Molinari

Abstract: As a consequence of missing data on tests for infection and imperfect accuracy of tests, reported rates of population infection by the SARS CoV-2 virus are lower than actual rates of infection. Hence, reported rates of severe illness conditional on infection are higher than actual rates. Understanding the time path of the COVID-19 pandemic has been hampered by the absence of bounds on infection ra… ▽ More As a consequence of missing data on tests for infection and imperfect accuracy of tests, reported rates of population infection by the SARS CoV-2 virus are lower than actual rates of infection. Hence, reported rates of severe illness conditional on infection are higher than actual rates. Understanding the time path of the COVID-19 pandemic has been hampered by the absence of bounds on infection rates that are credible and informative. This paper explains the logical problem of bounding these rates and reports illustrative findings, using data from Illinois, New York, and Italy. We combine the data with assumptions on the infection rate in the untested population and on the accuracy of the tests that appear credible in the current context. We find that the infection rate might be substantially higher than reported. We also find that the infection fatality rate in Italy is substantially lower than reported. △ Less

Submitted 13 April, 2020; originally announced April 2020.

Econometrics For Decision Making: Building Foundations Sketched By Haavelmo And Wald

Authors: Charles F. Manski

Abstract: Haavelmo (1944) proposed a probabilistic structure for econometric modeling, aiming to make econometrics useful for decision making. His fundamental contribution has become thoroughly embedded in subsequent econometric research, yet it could not answer all the deep issues that the author raised. Notably, Haavelmo struggled to formalize the implications for decision making of the fact that models c… ▽ More Haavelmo (1944) proposed a probabilistic structure for econometric modeling, aiming to make econometrics useful for decision making. His fundamental contribution has become thoroughly embedded in subsequent econometric research, yet it could not answer all the deep issues that the author raised. Notably, Haavelmo struggled to formalize the implications for decision making of the fact that models can at most approximate actuality. In the same period, Wald (1939, 1945) initiated his own seminal development of statistical decision theory. Haavelmo favorably cited Wald, but econometrics did not embrace statistical decision theory. Instead, it focused on study of identification, estimation, and statistical inference. This paper proposes statistical decision theory as a framework for evaluation of the performance of models in decision making. I particularly consider the common practice of as-if optimization: specification of a model, point estimation of its parameters, and use of the point estimate to make a decision that would be optimal if the estimate were accurate. A central theme is that one should evaluate as-if optimization or any other model-based decision rule by its performance across the state space, listing all states of nature that one believes feasible, not across the model space. I apply the theme to prediction and treatment choice. Statistical decision theory is conceptually simple, but application is often challenging. Advancement of computation is the primary task to continue building the foundations sketched by Haavelmo and Wald. △ Less

Submitted 22 February, 2021; v1 submitted 17 December, 2019; originally announced December 2019.

Comments: arXiv admin note: substantial text overlap with arXiv:1909.06853

Statistical inference for statistical decisions

Authors: Charles F. Manski

Abstract: The Wald development of statistical decision theory addresses decision making with sample data. Wald's concept of a statistical decision function (SDF) embraces all mappings of the form [data -> decision]. An SDF need not perform statistical inference; that is, it need not use data to draw conclusions about the true state of nature. Inference-based SDFs have the sequential form [data -> inference… ▽ More The Wald development of statistical decision theory addresses decision making with sample data. Wald's concept of a statistical decision function (SDF) embraces all mappings of the form [data -> decision]. An SDF need not perform statistical inference; that is, it need not use data to draw conclusions about the true state of nature. Inference-based SDFs have the sequential form [data -> inference -> decision]. This paper motivates inference-based SDFs as practical procedures for decision making that may accomplish some of what Wald envisioned. The paper first addresses binary choice problems, where all SDFs may be viewed as hypothesis tests. It next considers as-if optimization, which uses a point estimate of the true state as if the estimate were accurate. It then extends this idea to as-if maximin and minimax-regret decisions, which use point estimates of some features of the true state as if they were accurate. The paper primarily uses finite-sample maximum regret to evaluate the performance of inference-based SDFs. To illustrate abstract ideas, it presents specific findings concerning treatment choice and point prediction with sample data. △ Less

Submitted 15 September, 2019; originally announced September 2019.