-
Confidence Intervals for Selected Parameters
Authors:
Yoav Benjamini,
Yotam Hechtlinger,
Philip B. Stark
Abstract:
Practical or scientific considerations often lead to selecting a subset of parameters as ``important.'' Inferences about those parameters often are based on the same data used to select them in the first place. That can make the reported uncertainties deceptively optimistic: confidence intervals that ignore selection generally have less than their nominal coverage probability. Controlling the prob…
▽ More
Practical or scientific considerations often lead to selecting a subset of parameters as ``important.'' Inferences about those parameters often are based on the same data used to select them in the first place. That can make the reported uncertainties deceptively optimistic: confidence intervals that ignore selection generally have less than their nominal coverage probability. Controlling the probability that one or more intervals for selected parameters do not cover---the ``simultaneous over the selected'' (SoS) error rate---is crucial in many scientific problems. Intervals that control the SoS error rate can be constructed in ways that take advantage of knowledge of the selection rule. We construct SoS-controlling confidence intervals for parameters deemed the most ``important'' $k$ of $m$ shift parameters because they are estimated (by independent estimators) to be the largest. The new intervals improve substantially over Šidák intervals when $k$ is small compared to $m$, and approach the standard Bonferroni-corrected intervals when $k \approx m$. Standard, unadjusted confidence intervals for location parameters have the correct coverage probability for $k=1$, $m=2$ if, when the true parameters are zero, the estimators are exchangeable and symmetric.
△ Less
Submitted 2 June, 2019;
originally announced June 2019.
-
Another Argument in Favour of Wilcoxon's Signed Rank Test
Authors:
Jonathan Rosenblatt,
Yoav Benjamini
Abstract:
The Wilcoxon Signed Rank test is typically called upon when testing whether a symmetric distribution has a specified centre and the Gaussianity is in question. As with all insurance policies it comes with a cost, even if small, in terms of power versus a t-test, when the distribution is indeed Gaussian. In this note we further show that even when the distribution tested is Gaussian there need not…
▽ More
The Wilcoxon Signed Rank test is typically called upon when testing whether a symmetric distribution has a specified centre and the Gaussianity is in question. As with all insurance policies it comes with a cost, even if small, in terms of power versus a t-test, when the distribution is indeed Gaussian. In this note we further show that even when the distribution tested is Gaussian there need not be power loss at all, if the alternative is of a mixture type rather than a shift. The signed rank test may turn out to be more powerful than the t-test, and the supposedly conservative strategy, might actually be the more powerful one. Drug testing and functional magnetic imaging are two such scenarios.
Wilcoxon' signed rank test will typically be called upon by a researcher when testing for the location of a single population, using a small sample and Gaussianity is dubious. As all insurance policies, it will come with a cost-- power. It is well known, that under a Gaussian setup, the signed rank test is less powerful than, say, a t-test. The works of Pitman and others have reassured us that this power loss is surprisingly small. In this note we argue that the power loss might actually be smaller than typically assumed. In particular, if the deviation from the null Gaussian distribution is of a mixture type and not a shift type, the signed rank test is no longer dominated by the t-test and can actually be more powerful.
△ Less
Submitted 21 November, 2013;
originally announced November 2013.
-
Adjusting for selection bias in testing multiple families of hypotheses
Authors:
Yoav Benjamini,
Marina Bogomolov
Abstract:
In many large multiple testing problems the hypotheses are divided into families. Given the data, families with evidence for true discoveries are selected, and hypotheses within them are tested. Neither controlling the error-rate in each family separately nor controlling the error-rate over all hypotheses together can assure that an error-rate is controlled in the selected families. We formulate t…
▽ More
In many large multiple testing problems the hypotheses are divided into families. Given the data, families with evidence for true discoveries are selected, and hypotheses within them are tested. Neither controlling the error-rate in each family separately nor controlling the error-rate over all hypotheses together can assure that an error-rate is controlled in the selected families. We formulate this concern about selective inference in its generality, for a very wide class of error-rates and for any selection criterion, and present an adjustment of the testing level inside the selected families that retains the average error-rate over the selected families.
△ Less
Submitted 18 June, 2011;
originally announced June 2011.
-
An adaptive step-down procedure with proven FDR control under independence
Authors:
Yulia Gavrilov,
Yoav Benjamini,
Sanat K. Sarkar
Abstract:
In this work we study an adaptive step-down procedure for testing $m$ hypotheses. It stems from the repeated use of the false discovery rate controlling the linear step-up procedure (sometimes called BH), and makes use of the critical constants $iq/[(m+1-i(1-q)]$, $i=1,...,m$. Motivated by its success as a model selection procedure, as well as by its asymptotic optimality, we are interested in i…
▽ More
In this work we study an adaptive step-down procedure for testing $m$ hypotheses. It stems from the repeated use of the false discovery rate controlling the linear step-up procedure (sometimes called BH), and makes use of the critical constants $iq/[(m+1-i(1-q)]$, $i=1,...,m$. Motivated by its success as a model selection procedure, as well as by its asymptotic optimality, we are interested in its false discovery rate (FDR) controlling properties for a finite number of hypotheses. We prove this step-down procedure controls the FDR at level $q$ for independent test statistics. We then numerically compare it with two other procedures with proven FDR control under independence, both in terms of power under independence and FDR control under positive dependence.
△ Less
Submitted 31 March, 2009;
originally announced March 2009.
-
Adapting to Unknown Sparsity by controlling the False Discovery Rate
Authors:
Felix Abramovich,
Yoav Benjamini,
David L. Donoho,
Iain M. Johnstone
Abstract:
We attempt to recover an $n$-dimensional vector observed in white noise, where $n$ is large and the vector is known to be sparse, but the degree of sparsity is unknown. We consider three different ways of defining sparsity of a vector: using the fraction of nonzero terms; imposing power-law decay bounds on the ordered entries; and controlling the $\ell_p$ norm for $p$ small. We obtain a procedur…
▽ More
We attempt to recover an $n$-dimensional vector observed in white noise, where $n$ is large and the vector is known to be sparse, but the degree of sparsity is unknown. We consider three different ways of defining sparsity of a vector: using the fraction of nonzero terms; imposing power-law decay bounds on the ordered entries; and controlling the $\ell_p$ norm for $p$ small. We obtain a procedure which is asymptotically minimax for $\ell^r$ loss, simultaneously throughout a range of such sparsity classes.
The optimal procedure is a data-adaptive thresholding scheme, driven by control of the {\it False Discovery Rate} (FDR). FDR control is a relatively recent innovation in simultaneous testing, ensuring that at most a certain fraction of the rejected null hypotheses will correspond to false rejections.
In our treatment, the FDR control parameter $q_n$ also plays a determining role in asymptotic minimaxity. If $q = \lim q_n \in [0,1/2]$ and also $q_n > γ/\log(n)$ we get sharp asymptotic minimaxity, simultaneously, over a wide range of sparse parameter spaces and loss functions. On the other hand, $ q = \lim q_n \in (1/2,1]$, forces the risk to exceed the minimax risk by a factor growing with $q$.
To our knowledge, this relation between ideas in simultaneous inference and asymptotic decision theory is new.
Our work provides a new perspective on a class of model selection rules which has been introduced recently by several authors. These new rules impose complexity penalization of the form $2 \cdot \log({potential model size} / {actual model size})$. We exhibit a close connection with FDR-controlling procedures under stringent control of the false discovery rate.
△ Less
Submitted 18 May, 2005;
originally announced May 2005.
