-
Bagging cross-validated bandwidths with application to Big Data
Authors:
Daniel Barreiro-Ures,
Ricardo Cao,
Mario Francisco Fernández,
Jeffrey D. Hart
Abstract:
Hall and Robinson (2009) proposed and analyzed the use of bagged cross-validation to choose the bandwidth of a kernel density estimator. They established that bagging greatly reduces the noise inherent in ordinary cross-validation, and hence leads to a more efficient bandwidth selector. The asymptotic theory of Hall and Robinson (2009) assumes that $N$, the number of bagged subsamples, is…
▽ More
Hall and Robinson (2009) proposed and analyzed the use of bagged cross-validation to choose the bandwidth of a kernel density estimator. They established that bagging greatly reduces the noise inherent in ordinary cross-validation, and hence leads to a more efficient bandwidth selector. The asymptotic theory of Hall and Robinson (2009) assumes that $N$, the number of bagged subsamples, is $\infty$. We expand upon their theoretical results by allowing $N$ to be finite, as it is in practice. Our results indicate an important difference in the rate of convergence of the bagged cross-validation bandwidth for the cases $N=\infty$ and $N<\infty$. Simulations quantify the improvement in statistical efficiency and computational speed that can result from using bagged cross-validation as opposed to a binned implementation of ordinary cross-validation. The performance of thebagged bandwidth is also illustrated on a real, very large, data set. Finally, a byproduct of our study is the correction of errors appearing in the Hall and Robinson (2009) expression for the asymptotic mean squared error of the bagging selector.
△ Less
Submitted 31 January, 2024;
originally announced January 2024.
-
Screening Methods for Classification Based on Non-parametric Bayesian Tests
Authors:
Naveed Merchant,
Jeffrey D. Hart
Abstract:
Feature or variable selection is a problem inherent to large data sets. While many methods have been proposed to deal with this problem, some can scale poorly with the number of predictors in a data set. Screening methods scale linearly with the number of predictors by checking each predictor one at a time, and are a tool used to decrease the number of variables to consider before further analysis…
▽ More
Feature or variable selection is a problem inherent to large data sets. While many methods have been proposed to deal with this problem, some can scale poorly with the number of predictors in a data set. Screening methods scale linearly with the number of predictors by checking each predictor one at a time, and are a tool used to decrease the number of variables to consider before further analysis or variable selection. For classification, there is a variety of techniques. There are parametric based screening tests, such as t-test or SIS based screening, and non-parametric based screening tests, such as Kolmogorov distance based screening, and MV-SIS. We propose a method for variable screening that uses Bayesian-motivated tests, compare it to SIS based screening, and provide example applications of the method on simulated and real data. It is shown that our screening method can lead to improvements in classification rate. This is so even when our method is used in conjunction with a classifier, such as DART, which is designed to select a sparse subset of variables. Finally, we propose a classifier based on kernel density estimates that in some cases can produce dramatic improvements in classification rates relative to DART.
△ Less
Submitted 5 January, 2023;
originally announced January 2023.
-
Estimating the Mean and Variance of a High-dimensional Normal Distribution Using a Mixture Prior
Authors:
Shyamalendu Sinha,
Jeffrey D. Hart
Abstract:
This paper provides a framework for estimating the mean and variance of a high-dimensional normal density. The main setting considered is a fixed number of vector following a high-dimensional normal distribution with unknown mean and diagonal covariance matrix. The diagonal covariance matrix can be known or unknown. If the covariance matrix is unknown, the sample size can be as small as $2$. The p…
▽ More
This paper provides a framework for estimating the mean and variance of a high-dimensional normal density. The main setting considered is a fixed number of vector following a high-dimensional normal distribution with unknown mean and diagonal covariance matrix. The diagonal covariance matrix can be known or unknown. If the covariance matrix is unknown, the sample size can be as small as $2$. The proposed estimator is based on the idea that the unobserved pairs of mean and variance for each dimension are drawn from an unknown bivariate distribution, which we model as a mixture of normal-inverse gammas. The mixture of normal-inverse gamma distributions provides advantages over more traditional empirical Bayes methods, which are based on a normal-normal model. When fitting a mixture model, we are essentially clustering the unobserved mean and variance pairs for each dimension into different groups, with each group having a different normal-inverse gamma distribution. The proposed estimator of each mean is the posterior mean of shrinkage estimates, each of which shrinks a sample mean towards a different component of the mixture distribution. Similarly, the proposed estimator of variance has an analogous interpretation in terms of sample variances and components of the mixture distribution. If diagonal covariance matrix is known, then the sample size can be as small as $1$, and we treat the pairs of known variance and unknown mean for each dimension as random observations coming from a flexible mixture of normal-inverse gamma distributions.
△ Less
Submitted 15 November, 2018;
originally announced November 2018.
-
Partitioned Cross-Validation for Divide-and-Conquer Density Estimation
Authors:
Anirban Bhattacharya,
Jeffrey D. Hart
Abstract:
We present an efficient method to estimate cross-validation bandwidth parameters for kernel density estimation in very large datasets where ordinary cross-validation is rendered highly inefficient, both statistically and computationally. Our approach relies on calculating multiple cross-validation bandwidths on partitions of the data, followed by suitable scaling and averaging to return a partitio…
▽ More
We present an efficient method to estimate cross-validation bandwidth parameters for kernel density estimation in very large datasets where ordinary cross-validation is rendered highly inefficient, both statistically and computationally. Our approach relies on calculating multiple cross-validation bandwidths on partitions of the data, followed by suitable scaling and averaging to return a partitioned cross-validation bandwidth for the entire dataset. The partitioned cross-validation approach produces substantial computational gains over ordinary cross-validation. We additionally show that partitioned cross-validation can be statistically efficient compared to ordinary cross-validation. We derive analytic expressions for the asymptotically optimal number of partitions and study its finite sample accuracy through a detailed simulation study. We additionally propose a permuted version of partitioned cross-validation which attains even higher efficiency. Theoretical properties of the estimators are studied and the methodology is applied to the Higgs Boson dataset with 11 million observations
△ Less
Submitted 31 August, 2016;
originally announced September 2016.
-
Theoretical Properties and Practical Performance of Fully Robust One-Sided Cross-Validation
Authors:
Olga Y. Savchuk,
Jeffrey D. Hart
Abstract:
Fully robust OSCV is a modification of the OSCV method that produces consistent bandwidth in the cases of smooth and nonsmooth regression functions. The current implementation of the method uses the kernel $H_I$ that is almost indistinguishable from the Gaussian kernel on the interval $[-4,4]$, but has negative tails. The theoretical properties and practical performances of the $H_I$- and $φ$-base…
▽ More
Fully robust OSCV is a modification of the OSCV method that produces consistent bandwidth in the cases of smooth and nonsmooth regression functions. The current implementation of the method uses the kernel $H_I$ that is almost indistinguishable from the Gaussian kernel on the interval $[-4,4]$, but has negative tails. The theoretical properties and practical performances of the $H_I$- and $φ$-based OSCV versions are compared. The kernel $H_I$ tends to produce too low bandwidths in the smooth case. The $H_I$-based OSCV curves are shown to have wiggles appearing in the neighborhood of zero. The kernel $H_I$ uncovers sensitivity of the OSCV method to a tiny modification of the kernel used for the cross-validation purposes. The recently found robust bimodal kernels tend to produce OSCV curves with multiple local minima. The problem of finding a robust unimodal nonnegative kernel remains open.
△ Less
Submitted 26 February, 2016;
originally announced February 2016.
-
Empirical study of indirect cross-validation
Authors:
Olga Y. Savchuk,
Jeffrey D. Hart,
Simon J. Sheather
Abstract:
In this paper we provide insight into the empirical properties of indirect cross-validation (ICV), a new method of bandwidth selection for kernel density estimators. First, we describe the method and report on the theoretical results used to develop a practical-purpose model for certain ICV parameters. Next, we provide a detailed description of a numerical study which shows that the ICV method u…
▽ More
In this paper we provide insight into the empirical properties of indirect cross-validation (ICV), a new method of bandwidth selection for kernel density estimators. First, we describe the method and report on the theoretical results used to develop a practical-purpose model for certain ICV parameters. Next, we provide a detailed description of a numerical study which shows that the ICV method usually outperforms least squares cross-validation (LSCV) in finite samples. One of the major advantages of ICV is its increased stability compared to LSCV. Two real data examples show the benefit of using both ICV and a local version of ICV.
△ Less
Submitted 29 November, 2008;
originally announced December 2008.
-
Indirect Cross-validation for Density Estimation
Authors:
Olga Y. Savchuk,
Jeffrey D. Hart,
Simon J. Sheather
Abstract:
A new method of bandwidth selection for kernel density estimators is proposed. The method, termed indirect cross-validation, or ICV, makes use of so-called selection kernels. Least squares cross-validation (LSCV) is used to select the bandwidth of a selection-kernel estimator, and this bandwidth is appropriately rescaled for use in a Gaussian kernel estimator. The proposed selection kernels are…
▽ More
A new method of bandwidth selection for kernel density estimators is proposed. The method, termed indirect cross-validation, or ICV, makes use of so-called selection kernels. Least squares cross-validation (LSCV) is used to select the bandwidth of a selection-kernel estimator, and this bandwidth is appropriately rescaled for use in a Gaussian kernel estimator. The proposed selection kernels are linear combinations of two Gaussian kernels, and need not be unimodal or positive. Theory is developed showing that the relative error of ICV bandwidths can converge to 0 at a rate of $n^{-1/4}$, which is substantially better than the $n^{-1/10}$ rate of LSCV. Interestingly, the selection kernels that are best for purposes of bandwidth selection are very poor if used to actually estimate the density function. This property appears to be part of the larger and well-documented paradox to the effect that "the harder the estimation problem, the better cross-validation performs." The ICV method uniformly outperforms LSCV in a simulation study, a real data example, and a simulated example in which bandwidths are chosen locally.
△ Less
Submitted 29 November, 2008;
originally announced December 2008.
