Bootstrap: the issue of overfitting

Question

Suppose one performs the so-called non-parametric bootstrap by drawing $B$ samples of size $n$ each from the original $n$ observations with replacement. I believe this procedure is equivalent to estimating the cumulative distribution function by the empirical cdf:

http://en.wikipedia.org/wiki/Empirical_distribution_function

and then obtaining the bootstrap samples by simulating $n$ observations from the estimated cdf $B$ times in a row.

If I am right in this, then one has to address the issue of overfitting, because the empirical cdf has about N parameters. Of course, asymptotically it converges to the population cdf, but what about finite samples? E.g. if I were to tell you that I have 100 observations and I am going to estimate the cdf as $N(\mu, \sigma^2)$ with two parameters, you wouldn't be alarmed. However, if the number of parameters were to go up to 100, it wouldn't seem reasonable at all.

Likewise, when one employs a standard multiple linear regression, the distribution of the error term is estimated as $N(0, \sigma^2)$. If one decides to switch to bootstrapping the residuals, he has to realize that now there are about $n$ parameters used just to handle the error term distribution.

Could you please direct me to some sources that address this issue explicitly, or tell me why it's not an issue if you think I got it wrong.

Answer 1

i am not completely sure i understand your question right... i am assuming you are interested in the order of convergence?

because the empirical cdf has about N parameters. Of course, asymptotically it converges to the population cdf, but what about finite samples?

Have you read any of the basics on bootstrap theory? The Problem is that it gets pretty wild (mathematically) pretty quickly.

Anyway, i recommend having a look at

van der Vaart "Asymptotic Statistics" chapter 23.

Hall "Bootstrap and Edgeworth expansions" (lengthy but concise and less handwaving than van der Vaart i'd say)

for the basics.

Chernick "Bootstrap Methods" is more aimed at users rather than mathematicians but has a section on "where bootstrap fails".

The classical Efron/Tibshirani has little on why bootstrap actually works...

Answer 2

Janssen and Pauls showed that bootstrapping a statistic works asymptotically, iff a central limit theorem could also have been applied. So if you compare estimating the parameters of a $\mathcal{N}(\mu,\sigma^2)$ distribution as the distribution of the statistic and estimating the statistic's distribution via bootstrap hits the point.

Intuitively, bootstrapping from finite samples underestimates heavy tails of the underlying distribution. That's clear, since finite samples have a finite range, even if their true distribution's range is infinite or, even worse, has heavy tails. So the bootstrap statistic's behaviour will never be as "wild" as the original statistic. So similar to avoiding overfitting due to too many parameters in (parametric) regression, we could avoid overfitting by using the few-parameter normal distribution.

Edit responding the comments: Remember you don't need the bootstrap to estimate the cdf. You usually use the bootstrap to get the distribution (in the broadest sense including quantiles, moments, whatever needed) of some statistic. So you don't necessarily have an overfitting problem (in terms of "the estimation due to my finite data looks too nice comparing to what I should see with the true wild distribution"). But as it turned out (by the cited paper and by Frank Harrel's comment below), getting such an overfitting problem is linked to problems with parametric estimation of the same statistics.

So as your question implied, bootstrapping is not a panacea against problems with parametric estimation. The hope that the bootstrap would help with parameter problems by controlling the whole distribution is spurious.

Answer 3

One source of intuition might be to compare rates of convergence for parametric CDFs vs ECDFs, for iid data.

By DKW, the empirical CDF converges to the true CDF at a $n^{-1/2}$ rate (not just at one point, but the supremum of the absolute difference over the whole domain of the CDFs): https://en.wikipedia.org/wiki/Dvoretzky%E2%80%93Kiefer%E2%80%93Wolfowitz_inequality http://www.stat.cmu.edu/~larry/=stat705/Lecture12.pdf

And by Berry-Esseen, the CDF of a sampling distribution for a single mean converges to its Normal limit at a $n^{-1/2}$ rate: https://en.wikipedia.org/wiki/Berry%E2%80%93Esseen_theorem (This is not quite what we want---we're wondering about how the estimated parametric CDF of the data converges, not about the sampling distribution. But in the simplest ideal case, where the data are Normal and $\sigma$ is known and we just need to estimate $\mu$, I imagine the rates of convergence should be the same for the data's CDF as for the mean's CDF?)

So in a certain sense, the rate at which you need to acquire more samples is the same, whether you're estimating the CDF using an empirical CDF or whether you're estimating a parameter directly using a sample-mean-type estimator. This might help justify Frank Harrell's comment that "The number of effective parameters is not the same as the sample size."

Of course, that's not the whole story. Although the rates don't differ, the constants do. And there's much more to the nonparametric bootstrap than ECDFs---you still need to do things with the ECDF once you estimate it.