When is the bootstrap estimate of bias valid?

Question

It is often claimed that bootstrapping can provide an estimate of the bias in an estimator.

If $\hat t$ is the estimate for some statistic, and $\tilde t_i$ are the bootstrap replicas (with $i\in\{1,\cdots,N\}$), then the bootstrap estimate of bias is \begin{equation} \mathrm{bias}_t \approx \frac{1}{N}\sum_i \tilde{t}_i-\hat t \end{equation} which seems extremely simple and powerful, to the point of being unsettling.

I can't get my head around how this is possible without having an unbiased estimator of the statistic already. For example, if my estimator simply returns a constant that is independent of the observations, the above estimate of bias is clearly invalid.

Although this example is pathological, I can't see what are the reasonable assumptions about the estimator and the distributions that will guarantee that the bootstrap estimate is reasonable.

I tried reading the formal references, but I am not a statistician nor a mathematician, so nothing was clarified.

Can anyone provide a high level summary of when the estimate can be expected to be valid? If you know of good references on the subject that would also be great.

---

Edit:

Smoothness of the estimator is often quoted as a requirement for the bootstrap to work. Could it be that one also requires some sort of local invertibility of the transformation? The constant map clearly does not satisfy that.

Answer 1

I think your formula is wrong. The last $t$ should have a star rather than a hat: \begin{equation} \mathrm{bias}_t \approx \frac{1}{N}\sum_i \tilde{t}_i- t^* \end{equation}

You want to use the actual statistic evaluated on the empirical distribution (this is often easy, since the original sample is a finite set), rather than the estimate. In some cases, these may be the same (for example, the empirical mean is the same as the sample mean), but they won't be in general. You gave one case where they are different, but a less pathological example is the usual unbiased estimator for variance, which is not the same as the population variance when applied to a finite distribution.

If the statistic $t$ doesn't make sense on the empirical distribution (for example, if it assumes a continuous distribution), then you shouldn't use vanilla bootstrapping. You can replace the empirical distribution with a kernel density estimate (smooth bootstrap), or if you know that the original distribution lies in some particular family, you can replace the empirical distribution with the maximum likely estimate from that family (parametric bootstrap).

TL/DR: The bootstrap method is not magical. To get an unbiased estimate of the bias, you need to be able to compute the parameter of interest exactly on a finite distribution.

Answer 2

The problem you describe is a problem of interpretation, not one of validity. The bootstrap bias estimate for your constant estimator isn't invalid, it is in fact perfect.

The bootstrap estimate of bias is between an estimator $\hat\theta = s(x)$ and a parameter $\theta = t(F),$ where $F$ is some unknown distribution and $x$ a sample from $F$. The function $t(F)$ is something you could in principle calculate if you had the population at hand. Some times we take $s(x) = t(\hat F),$ the plug-in estimate of $t(F)$ using the empirical distribution $\hat F$ in the place of $F$. This is presumably what you describe above. In all cases the bootstrap estimate of bias is $$ \mathrm{bias}_{\hat F} = E_{\hat F}[s(x^*)] - t(\hat F), $$ where $x^*$ are bootstrap samples from $x$.

The constant $c$ is a perfect plug-in estimate for that same constant: The population is $\sim F$ and the sample $\sim \hat F$, the empirical distribution, which approximates $F$. If you could evaluate $t(F) = c$, you'd get $c$. When you compute the plug-in estimate $t(\hat F) = c$ you also get $c$. No bias, as you would expect.

A well-known case where there is a bias in the plug-in estimate $t(\hat F)$ is in estimating variance, hence Bessel's correction. Below I demonstrate this. The bootstrap bias estimate isn't too bad:

library(plyr)

n <- 20
data <- rnorm(n, 0, 1)

variance <- sum((data - mean(data))^2)/n

boots <- raply(1000, {
  data_b <- sample(data, n, replace=T)
  sum((data_b - mean(data_b))^2)/n
})

# estimated bias
mean(boots) - variance 
#> [1] -0.06504726

# true bias:
((n-1)/n)*1 -1
#> [1] -0.05

We could instead take $t(F)$ to be the population mean and $s(x) = c$, situation where in most cases there should be a clear bias:

library(plyr)

mu <- 3
a_constant <- 1

n <- 20
data <- rnorm(n, mu, 1)

boots <- raply(1000, {
  # not necessary as we will ignore the data, but let's do it on principle
  data_b <- sample(data, n, replace=T)

  a_constant
})

# estimated bias
mean(boots) - mean(data) 
#> [1] -1.964877

# true bias is clearly -2

Again the bootstrap estimate isn't too bad.

Answer 3

You make one mistake and maybe that is the reason it is confusing. You say:

if my estimator simply returns a constant that is independent of the observations, the above estimate of bias is clearly invalid

Bootstrap is not about how much your method is biased, but how much your results obtained by some function, given your data are biased.

If you choose appropriate statistical method for analyzing your data, and all the assumptions of this method are met, and you did your math correctly, then your statistical method should provide you the "best" possible estimate that can be obtained using your data.

The idea of bootstrap is to sample from your data the same way as you sampled your cases from the population - so it is a kind of replication of your sampling. This lets you to obtain approximate distribution (using Efrons words) of your value and hence to asses bias of your estimate.

However, what I argue is that your example is misleading and so it is not the best example for discussing bootstrap. Since there were misunderstandings on both sides, let me update my answer and write it in more formal way to illustrate my point.

Bias for $\hat{\theta}$ being estimate of true value $\theta$ is defined as:

$$\text{bias}(\hat{\theta}_n) = \mathbb{E}_\theta(\hat{\theta}_n) - \theta$$

where:

$$\hat{\theta}_n = g(x_1,x_2,...,x_n)$$

where $g(\cdot)$ is the estimator.

As Larry Wasserman notes in his book "All the Statistics":

A reasonable requirement for an estimator is that it should converge to the true parameter value as we collect more and more data. This requirement is quantified by the following definition:
6.7 Definition. A point estimator $\hat{\theta}_n$ of a parameter $\theta$ is consistent if $\hat{\theta}_n \overset{P}{\rightarrow} \theta$.

Constant estimator, being a constant function of $x$: $g(X) = \lambda$ does not meet this requirement since it is independent of data and growing number of observations would not make it approach the true value $\theta$ (unless by pure luck or having very solid a priori assumptions on $\lambda$ it is that $\lambda = \theta$).

Constant estimator does not meet the basic requirement for being a reasonable estimator and hence, it is impossible to estimate it's bias because $\hat{\theta}_n$ does not approach $\theta$ even with $n \rightarrow \infty$. It's impossible to do it with bootstrap and with any other method, so it's not a problem with bootstrap.

Answer 4

I find it useful to think about the bootstrap procedures in terms of the functionals of the distributions they operate on -- I gave an example in this answer to a different bootstrap question.

The estimate you gave is what it is -- an estimate. Nobody says it does not suffer from problems that statistical estimates may have. It will give you a non-zero estimate of bias for the sample mean, for instance, which we all know is unbiased to begin with. One problem with this bias estimator is that it suffers from sampling variability when the bootstrap is implemented as Monte Carlo rather than a complete enumeration of all possible subsamples (and nobody that that theoretical bootstrap in practice, anyway).

As such, a Monte Carlo implementation of the bootstrap is unfixable, and you have to use a different bootstrap scheme. Davison et. al. (1986) demonstrated how to create a different bootstrap scheme that restricts the random draws to produce balanced samples: if you create $B$ bootstrap replicates, then each of the original elements needs to be used exactly $B$ times for the first-order balance. (The second order balance that works better for the second moments of the estimands, is further discussed by Graham et. al. (1990).)