Showing 1–11 of 11 results for author: Horowitz, J

Deep Quantile Regression: Mitigating the Curse of Dimensionality Through Composition

Authors: Guohao Shen, Yuling Jiao, Yuanyuan Lin, Joel L. Horowitz, Jian Huang

Abstract: This paper considers the problem of nonparametric quantile regression under the assumption that the target conditional quantile function is a composition of a sequence of low-dimensional functions. We study the nonparametric quantile regression estimator using deep neural networks to approximate the target conditional quantile function. For convenience, we shall refer to such an estimator as a dee… ▽ More This paper considers the problem of nonparametric quantile regression under the assumption that the target conditional quantile function is a composition of a sequence of low-dimensional functions. We study the nonparametric quantile regression estimator using deep neural networks to approximate the target conditional quantile function. For convenience, we shall refer to such an estimator as a deep quantile regression (DQR) estimator. We show that the DQR estimator achieves the nonparametric optimal convergence rate up to a logarithmic factor determined by the intrinsic dimension of the underlying compositional structure of the conditional quantile function, not the ambient dimension of the predictor. Therefore, DQR is able to mitigate the curse of dimensionality under the assumption that the conditional quantile function has a compositional structure. To establish these results, we analyze the approximation error of a composite function by neural networks and show that the error rate only depends on the dimensions of the component functions. We apply our general results to several important statistical models often used in mitigating the curse of dimensionality, including the single index, the additive, the projection pursuit, the univariate composite, and the generalized hierarchical interaction models. We explicitly describe the prefactors in the error bounds in terms of the dimensionality of the data and show that the prefactors depends on the dimensionality linearly or quadratically in these models. We also conduct extensive numerical experiments to evaluate the effectiveness of DQR and demonstrate that it outperforms a kernel-based method for nonparametric quantile regression. △ Less

Submitted 1 August, 2021; v1 submitted 10 July, 2021; originally announced July 2021.

Comments: Guohao Shen and Yuling Jiao contributed equally to this work. Co-corresponding authors: Yuanyuan Lin (email: [email protected]) and Jian Huang (email: [email protected])

MSC Class: 62G05; 62G08; 68T07

Non-Asymptotic Inference in Instrumental Variables Estimation

Authors: Joel L. Horowitz

Abstract: This paper presents a simple method for carrying out inference in a wide variety of possibly nonlinear IV models under weak assumptions. The method is non-asymptotic in the sense that it provides a finite sample bound on the difference between the true and nominal probabilities of rejecting a correct null hypothesis. The method is a non-Studentized version of the Anderson-Rubin test but is motivat… ▽ More This paper presents a simple method for carrying out inference in a wide variety of possibly nonlinear IV models under weak assumptions. The method is non-asymptotic in the sense that it provides a finite sample bound on the difference between the true and nominal probabilities of rejecting a correct null hypothesis. The method is a non-Studentized version of the Anderson-Rubin test but is motivated and analyzed differently. In contrast to the conventional Anderson-Rubin test, the method proposed here does not require restrictive distributional assumptions, linearity of the estimated model, or simultaneous equations. Nor does it require knowledge of whether the instruments are strong or weak. It does not require testing or estimating the strength of the instruments. The method can be applied to quantile IV models that may be nonlinear and can be used to test a parametric IV model against a nonparametric alternative. The results presented here hold in finite samples, regardless of the strength of the instruments. △ Less

Submitted 10 September, 2018; originally announced September 2018.

Comments: 33 pages, 5 tables

A simple bootstrap method for constructing nonparametric confidence bands for functions

Authors: Peter Hall, Joel Horowitz

Abstract: Standard approaches to constructing nonparametric confidence bands for functions are frustrated by the impact of bias, which generally is not estimated consistently when using the bootstrap and conventionally smoothed function estimators. To overcome this problem it is common practice to either undersmooth, so as to reduce the impact of bias, or oversmooth, and thereby introduce an explicit or imp… ▽ More Standard approaches to constructing nonparametric confidence bands for functions are frustrated by the impact of bias, which generally is not estimated consistently when using the bootstrap and conventionally smoothed function estimators. To overcome this problem it is common practice to either undersmooth, so as to reduce the impact of bias, or oversmooth, and thereby introduce an explicit or implicit bias estimator. However, these approaches, and others based on nonstandard smoothing methods, complicate the process of inference, for example, by requiring the choice of new, unconventional smoothing parameters and, in the case of undersmoothing, producing relatively wide bands. In this paper we suggest a new approach, which exploits to our advantage one of the difficulties that, in the past, has prevented an attractive solution to the problem - the fact that the standard bootstrap bias estimator suffers from relatively high-frequency stochastic error. The high frequency, together with a technique based on quantiles, can be exploited to dampen down the stochastic error term, leading to relatively narrow, simple-to-construct confidence bands. △ Less

Submitted 19 September, 2013; originally announced September 2013.

Comments: Published in at http://dx.doi.org/10.1214/13-AOS1137 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-AOS-AOS1137

Journal ref: Annals of Statistics 2013, Vol. 41, No. 4, 1892-1921

Variable selection in nonparametric additive models

Authors: Jian Huang, Joel L. Horowitz, Fengrong Wei

Abstract: We consider a nonparametric additive model of a conditional mean function in which the number of variables and additive components may be larger than the sample size but the number of nonzero additive components is "small" relative to the sample size. The statistical problem is to determine which additive components are nonzero. The additive components are approximated by truncated series expansio… ▽ More We consider a nonparametric additive model of a conditional mean function in which the number of variables and additive components may be larger than the sample size but the number of nonzero additive components is "small" relative to the sample size. The statistical problem is to determine which additive components are nonzero. The additive components are approximated by truncated series expansions with B-spline bases. With this approximation, the problem of component selection becomes that of selecting the groups of coefficients in the expansion. We apply the adaptive group Lasso to select nonzero components, using the group Lasso to obtain an initial estimator and reduce the dimension of the problem. We give conditions under which the group Lasso selects a model whose number of components is comparable with the underlying model, and the adaptive group Lasso selects the nonzero components correctly with probability approaching one as the sample size increases and achieves the optimal rate of convergence. The results of Monte Carlo experiments show that the adaptive group Lasso procedure works well with samples of moderate size. A data example is used to illustrate the application of the proposed method. △ Less

Submitted 20 October, 2010; originally announced October 2010.

Comments: Published in at http://dx.doi.org/10.1214/09-AOS781 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-AOS-AOS781

Journal ref: Annals of Statistics 2010, Vol. 38, No. 4, 2282-2313

Asymptotic properties of bridge estimators in sparse high-dimensional regression models

Authors: Jian Huang, Joel L. Horowitz, Shuangge Ma

Abstract: We study the asymptotic properties of bridge estimators in sparse, high-dimensional, linear regression models when the number of covariates may increase to infinity with the sample size. We are particularly interested in the use of bridge estimators to distinguish between covariates whose coefficients are zero and covariates whose coefficients are nonzero. We show that under appropriate conditio… ▽ More We study the asymptotic properties of bridge estimators in sparse, high-dimensional, linear regression models when the number of covariates may increase to infinity with the sample size. We are particularly interested in the use of bridge estimators to distinguish between covariates whose coefficients are zero and covariates whose coefficients are nonzero. We show that under appropriate conditions, bridge estimators correctly select covariates with nonzero coefficients with probability converging to one and that the estimators of nonzero coefficients have the same asymptotic distribution that they would have if the zero coefficients were known in advance. Thus, bridge estimators have an oracle property in the sense of Fan and Li [J. Amer. Statist. Assoc. 96 (2001) 1348--1360] and Fan and Peng [Ann. Statist. 32 (2004) 928--961]. In general, the oracle property holds only if the number of covariates is smaller than the sample size. However, under a partial orthogonality condition in which the covariates of the zero coefficients are uncorrelated or weakly correlated with the covariates of nonzero coefficients, we show that marginal bridge estimators can correctly distinguish between covariates with nonzero and zero coefficients with probability converging to one even when the number of covariates is greater than the sample size. △ Less

Submitted 4 April, 2008; originally announced April 2008.

Comments: Published in at http://dx.doi.org/10.1214/009053607000000875 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-AOS-AOS0324 MSC Class: 62J05; 62J07 (Primary) 62E20; 60F05 (Secondary)

Journal ref: Annals of Statistics 2008, Vol. 36, No. 2, 587-613

Rate-optimal estimation for a general class of nonparametric regression models with unknown link functions

Authors: Joel L. Horowitz, Enno Mammen

Abstract: This paper discusses a nonparametric regression model that naturally generalizes neural network models. The model is based on a finite number of one-dimensional transformations and can be estimated with a one-dimensional rate of convergence. The model contains the generalized additive model with unknown link function as a special case. For this case, it is shown that the additive components and… ▽ More This paper discusses a nonparametric regression model that naturally generalizes neural network models. The model is based on a finite number of one-dimensional transformations and can be estimated with a one-dimensional rate of convergence. The model contains the generalized additive model with unknown link function as a special case. For this case, it is shown that the additive components and link function can be estimated with the optimal rate by a smoothing spline that is the solution of a penalized least squares criterion. △ Less

Submitted 20 March, 2008; originally announced March 2008.

Comments: Published in at http://dx.doi.org/10.1214/009053607000000415 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-AOS-AOS0277 MSC Class: 62G08 (Primary) 62G20 (Secondary)

Journal ref: Annals of Statistics 2007, Vol. 35, No. 6, 2589-2619

Methodology and convergence rates for functional linear regression

Authors: Peter Hall, Joel L. Horowitz

Abstract: In functional linear regression, the slope ``parameter'' is a function. Therefore, in a nonparametric context, it is determined by an infinite number of unknowns. Its estimation involves solving an ill-posed problem and has points of contact with a range of methodologies, including statistical smoothing and deconvolution. The standard approach to estimating the slope function is based explicitly… ▽ More In functional linear regression, the slope ``parameter'' is a function. Therefore, in a nonparametric context, it is determined by an infinite number of unknowns. Its estimation involves solving an ill-posed problem and has points of contact with a range of methodologies, including statistical smoothing and deconvolution. The standard approach to estimating the slope function is based explicitly on functional principal components analysis and, consequently, on spectral decomposition in terms of eigenvalues and eigenfunctions. We discuss this approach in detail and show that in certain circumstances, optimal convergence rates are achieved by the PCA technique. An alternative approach based on quadratic regularisation is suggested and shown to have advantages from some points of view. △ Less

Submitted 3 August, 2007; originally announced August 2007.

Comments: Published at http://dx.doi.org/10.1214/009053606000000957 in the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-AOS-AOS0154 MSC Class: 62J05 (Primary) 62G20 (Secondary)

Journal ref: Annals of Statistics 2007, Vol. 35, No. 1, 70-91

Nonparametric methods for inference in the presence of instrumental variables

Authors: Peter Hall, Joel L. Horowitz

Abstract: We suggest two nonparametric approaches, based on kernel methods and orthogonal series to estimating regression functions in the presence of instrumental variables. For the first time in this class of problems, we derive optimal convergence rates, and show that they are attained by particular estimators. In the presence of instrumental variables the relation that identifies the regression functi… ▽ More We suggest two nonparametric approaches, based on kernel methods and orthogonal series to estimating regression functions in the presence of instrumental variables. For the first time in this class of problems, we derive optimal convergence rates, and show that they are attained by particular estimators. In the presence of instrumental variables the relation that identifies the regression function also defines an ill-posed inverse problem, the ``difficulty'' of which depends on eigenvalues of a certain integral operator which is determined by the joint density of endogenous and instrumental variables. We delineate the role played by problem difficulty in determining both the optimal convergence rate and the appropriate choice of smoothing parameter. △ Less

Submitted 6 March, 2006; originally announced March 2006.

Comments: Published at http://dx.doi.org/10.1214/009053605000000714 in the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-AOS-AOS0089 MSC Class: 62G08 (Primary) 62G20 (Secondary)

Journal ref: Annals of Statistics 2005, Vol. 33, No. 6, 2904-2929

Nonparametric Estimation of an Additive Model With a Link Function

Authors: Joel L. Horowitz, Enno Mammen

Abstract: This paper describes an estimator of the additive components of a nonparametric additive model with a known link function. When the additive components are twice continuously differentiable, the estimator is asymptotically normally distributed with a rate of convergence in probability of n^{-2/5}. This is true regardless of the (finite) dimension of the explanatory variable. Thus, in contrast to… ▽ More This paper describes an estimator of the additive components of a nonparametric additive model with a known link function. When the additive components are twice continuously differentiable, the estimator is asymptotically normally distributed with a rate of convergence in probability of n^{-2/5}. This is true regardless of the (finite) dimension of the explanatory variable. Thus, in contrast to the existing asymptotically normal estimator, the new estimator has no curse of dimensionality. Moreover, the estimator has an oracle property. The asymptotic distribution of each additive component is the same as it would be if the other components were known with certainty. △ Less

Submitted 30 August, 2005; originally announced August 2005.

Comments: Published at http://dx.doi.org/10.1214/009053604000000814 in the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-AOS-AOS240 MSC Class: 62G08 (Primary) 62G20 (Secondary)

Journal ref: Annals of Statistics 2004, Vol. 32, No. 6, 2412-2443

Moduli space of symplectic connections of Ricci type on $T^{2n}$; a formal approach

Authors: M. Cahen, S. Gutt, J. Horowitz, J. Rawnsley

Abstract: We consider analytic curves $\nabla^t$ of symplectic connections of Ricci type on the torus $T^{2n}$ with $\nabla^0$ the standard connection. We show, by a recursion argument, that if $\nabla^t$ is a formal curve of such connections then there exists a formal curve of symplectomorphisms $ψ_t$ such that $ψ_t\cdot\nabla^t$ is a formal curve of flat invariant symplectic connections and so… ▽ More We consider analytic curves $\nabla^t$ of symplectic connections of Ricci type on the torus $T^{2n}$ with $\nabla^0$ the standard connection. We show, by a recursion argument, that if $\nabla^t$ is a formal curve of such connections then there exists a formal curve of symplectomorphisms $ψ_t$ such that $ψ_t\cdot\nabla^t$ is a formal curve of flat invariant symplectic connections and so $\nabla^t$ is flat for all $t$. Applying this result to the Taylor series of the analytic curve, it means that analytic curves of symplectic connections of Ricci type starting at $\nabla^0$ are also flat. The group $G$ of symplectomorphisms of the torus $(T^{2n},ω)$ acts on the space $\E$ of symplectic connections which are of Ricci type. As a preliminary to studying the moduli space $\E/G$ we study the moduli of formal curves of connections under the action of formal curves of symplectomorphisms. △ Less

Submitted 8 February, 2002; v1 submitted 18 January, 2002; originally announced January 2002.

Comments: LaTeX, 19 pages. Removed claim to handle all symplectic structures on the torus

Homogeneous symplectic manifolds with Ricci-type curvature

Authors: M. Cahen, S. Gutt, J. Horowitz, J. Rawnsley

Abstract: We consider invariant symplectic connections $\nabla$ on homogeneous symplectic manifolds $(M,ω)$ with curvature of Ricci type. Such connections are solutions of a variational problem studied by Bourgeois and Cahen, and provide an integrable almost complex structure on the bundle of almost complex structures compatible with the symplectic structure. If $M$ is compact with finite fundamental grou… ▽ More We consider invariant symplectic connections $\nabla$ on homogeneous symplectic manifolds $(M,ω)$ with curvature of Ricci type. Such connections are solutions of a variational problem studied by Bourgeois and Cahen, and provide an integrable almost complex structure on the bundle of almost complex structures compatible with the symplectic structure. If $M$ is compact with finite fundamental group then $(M,ω)$ is symplectomorphic to $¶_n(\C)$ with a multiple of its Kähler form and $\nabla$ is affinely equivalent to the Levi-Civita connection. △ Less

Submitted 28 June, 2000; originally announced June 2000.

MSC Class: 53C05; 58C35; 53C57