When is the bootstrap estimate of bias valid?

It is often claimed that bootstrapping can provide an estimate of the bias in an estimator.

If $\hat t$ is the estimate for some statistic, and $\tilde t_i$ are the bootstrap replicas (with $i\in\{1,\cdots,N\}$), then the bootstrap estimate of bias is \begin{equation} \mathrm{bias}_t \approx \frac{1}{N}\sum_i \tilde{t}_i-\hat t \end{equation} which seems extremely simple and powerful, to the point of being unsettling.

I can't get my head around how this is possible without having an unbiased estimator of the statistic already. For example, if my estimator simply returns a constant that is independent of the observations, the above estimate of bias is clearly invalid.

Although this example is pathological, I can't see what are the reasonable assumptions about the estimator and the distributions that will guarantee that the bootstrap estimate is reasonable.

I tried reading the formal references, but I am not a statistician nor a mathematician, so nothing was clarified.

Can anyone provide a high level summary of when the estimate can be expected to be valid? If you know of good references on the subject that would also be great.

Edit:

Smoothness of the estimator is often quoted as a requirement for the bootstrap to work. Could it be that one also requires some sort of local invertibility of the transformation? The constant map clearly does not satisfy that.

• A constant estimator is an unbiased estimator of that constant so it is natural that the bootstrap estimator of the bias is zero. – Xi'an Dec 21 '14 at 11:51

You make one mistake and maybe that is the reason it is confusing. You say:

if my estimator simply returns a constant that is independent of the observations, the above estimate of bias is clearly invalid

Bootstrap is not about how much your method is biased, but how much your results obtained by some function, given your data are biased.

If you choose appropriate statistical method for analyzing your data, and all the assumptions of this method are met, and you did your math correctly, then your statistical method should provide you the "best" possible estimate that can be obtained using your data.

The idea of bootstrap is to sample from your data the same way as you sampled your cases from the population - so it is a kind of replication of your sampling. This lets you to obtain approximate distribution (using Efrons words) of your value and hence to asses bias of your estimate.

However, what I argue is that your example is misleading and so it is not the best example for discussing bootstrap. Since there were misunderstandings on both sides, let me update my answer and write it in more formal way to illustrate my point.

Bias for $\hat{\theta}$ being estimate of true value $\theta$ is defined as:

$$\text{bias}(\hat{\theta}_n) = \mathbb{E}_\theta(\hat{\theta}_n) - \theta$$

where:

$$\hat{\theta}_n = g(x_1,x_2,...,x_n)$$

where $g(\cdot)$ is the estimator.

As Larry Wasserman notes in his book "All the Statistics":

A reasonable requirement for an estimator is that it should converge to the true parameter value as we collect more and more data. This requirement is quantified by the following definition:
6.7 Definition. A point estimator $\hat{\theta}_n$ of a parameter $\theta$ is consistent if $\hat{\theta}_n \overset{P}{\rightarrow} \theta$.

Constant estimator, being a constant function of $x$: $g(X) = \lambda$ does not meet this requirement since it is independent of data and growing number of observations would not make it approach the true value $\theta$ (unless by pure luck or having very solid a priori assumptions on $\lambda$ it is that $\lambda = \theta$).

Constant estimator does not meet the basic requirement for being a reasonable estimator and hence, it is impossible to estimate it's bias because $\hat{\theta}_n$ does not approach $\theta$ even with $n \rightarrow \infty$. It's impossible to do it with bootstrap and with any other method, so it's not a problem with bootstrap.

• I am afraid this answer seems destined to sow confusion. A constant estimator is an estimator according to most definitions--and in some cases it is even an admissible one. Your question confounds sampling bias with estimation bias, which is bound to confuse almost all readers. Your paragraph about the "best possible estimate" is nice but it begs the essential question of how to measure "best." Bias is only one component of that (if at all). – whuber Dec 17 '14 at 20:39
• While I'm not qualified enough to answer OP, I'm afraid Whuber's got a point. Also, is it valid to call population mean an estimator? Related to the last sentence, I think boostrap provides an estimate of the bias of the estimator under analysis and not of sampling method. – mugen Dec 17 '14 at 20:58
• I understand that bootstrapping cannot detect systematic errors, but at least in some limit it is supposed to detect statistical bias. I suppose your point is about the subtlety in distinguishing between the two, but that is still unclear to me. You seem to be talking about a notion of bias that I never heard of -- not of the estimator, but of the data. What is the formal definition of this notion of bias? – Bootstrapped Dec 18 '14 at 14:25
• There is definitely a misunderstanding: Tim, you are not using "estimator"or "bias" in a way that is conventional for the context established in this question, whereas Bootstrapped is. Moreover, you are incorrect that the bootstrap can detect systematic errors and incorrect in equating those with "bias" in the context of estimation. There remain various errors in the answer, too. For instance, the bias of a constant estimator (equal, say, to $\lambda$) of a parameter $\theta$ is by definition $\lambda-\theta$. Please consult references. – whuber Dec 18 '14 at 17:36
• It's interesting that you bring up the issue of consistency in your edit. You might find it amusing--and maybe even a little thought-provoking--to contemplate the estimator $\hat\theta$ that equals $0$ provided $n\lt 10^{100}$ and otherwise is the maximum likelihood estimator. Although this is consistent, it suffers from the problem indicated by the O.P. Since this thread concerns characterizing conditions that would assure "the bootstrap estimate is reasonable," it would seem from this example that consistency is not among those conditions, nor is it even a relevant concept. – whuber Dec 19 '14 at 16:41

I think your formula is wrong. The last $t$ should have a star rather than a hat: \begin{equation} \mathrm{bias}_t \approx \frac{1}{N}\sum_i \tilde{t}_i- t^* \end{equation}

You want to use the actual statistic evaluated on the empirical distribution (this is often easy, since the original sample is a finite set), rather than the estimate. In some cases, these may be the same (for example, the empirical mean is the same as the sample mean), but they won't be in general. You gave one case where they are different, but a less pathological example is the usual unbiased estimator for variance, which is not the same as the population variance when applied to a finite distribution.

If the statistic $t$ doesn't make sense on the empirical distribution (for example, if it assumes a continuous distribution), then you shouldn't use vanilla bootstrapping. You can replace the empirical distribution with a kernel density estimate (smooth bootstrap), or if you know that the original distribution lies in some particular family, you can replace the empirical distribution with the maximum likely estimate from that family (parametric bootstrap).

TL/DR: The bootstrap method is not magical. To get an unbiased estimate of the bias, you need to be able to compute the parameter of interest exactly on a finite distribution.

• I am unsure about the meaning of your notation. According to these lecture notes by Pete Hall (UC Davis), these lecture notes by Cosma Shalizi (CMU), and this page of Efron's and Tibshirani's book seem to indicate that what I have it not wrong, just not fully general (i.e., I am using the plug in estimator here, but that is not necessary). – Bootstrapped Dec 18 '14 at 14:20
• Efron and Tibshirani give the same formula as me, with a different notation. Pete Hall seems to be making the assumption that $t^* = \hat t$: On page 11, he replaces $\theta(F_1)$ (which is what I called $t^*$ with $\hat \theta$ without comment. Cosma Shalizi's discussion of pivots in section 2.2 also seems to implicitly assume that $\hat t$ is the actual value of $t$ on the empirical distribution ($t^*$). I think all of your confusion is just caused by sloppiness in these lecture notes. – Evan Wright Dec 18 '14 at 15:25
• Fair enough, but I don't think the notation resolves the issue or addresses the question. In particular, I know the constant estimator has to break down (bootstrap is not magical). The example of the variance works even if we make the assumption that $t^*=\hat t$ (i.e., the bootstrap bias estimate works). What about other estimators for other statistics? What are sufficient conditions for the bootstrap bias estimate to work? How does the constant estimator violate these conditions? – Bootstrapped Dec 18 '14 at 18:43
• That's my point: this fixed version gives the right answer even for the constant estimator. Suppose you're trying to estimate the population mean, but you choose an estimator that just always guesses 0. Then $t^*$ will be the actual mean of the sample, rather than 0. So as $N \to \infty$, the bias estimate goes to minus the sample mean, which is reasonable and has expected value equal to the true bias. – Evan Wright Dec 18 '14 at 19:03
• Then it seems I don't quite understand the definition of $t^*$. The definition in Efron and Tibshirani (in the page I link to above) seems to imply that it is the plug in estimate based on the empirical distribution, but the operational meaning of that escaped me. Say I have some high dimensional data that I want to fit to some non-linear function, and I want to know if my estimate of the non-linear function parameters is biased or not. What is $t^*$ in this case? The definition of $\tilde t_i$ seems clear to me, but $t^*$ is nebulous. – Bootstrapped Dec 18 '14 at 20:08

The problem you describe is a problem of interpretation, not one of validity. The bootstrap bias estimate for your constant estimator isn't invalid, it is in fact perfect.

The bootstrap estimate of bias is between an estimator $\hat\theta = s(x)$ and a parameter $\theta = t(F),$ where $F$ is some unknown distribution and $x$ a sample from $F$. The function $t(F)$ is something you could in principle calculate if you had the population at hand. Some times we take $s(x) = t(\hat F),$ the plug-in estimate of $t(F)$ using the empirical distribution $\hat F$ in the place of $F$. This is presumably what you describe above. In all cases the bootstrap estimate of bias is $$\mathrm{bias}_{\hat F} = E_{\hat F}[s(x^*)] - t(\hat F),$$ where $x^*$ are bootstrap samples from $x$.

The constant $c$ is a perfect plug-in estimate for that same constant: The population is $\sim F$ and the sample $\sim \hat F$, the empirical distribution, which approximates $F$. If you could evaluate $t(F) = c$, you'd get $c$. When you compute the plug-in estimate $t(\hat F) = c$ you also get $c$. No bias, as you would expect.

A well-known case where there is a bias in the plug-in estimate $t(\hat F)$ is in estimating variance, hence Bessel's correction. Below I demonstrate this. The bootstrap bias estimate isn't too bad:

library(plyr)

n <- 20
data <- rnorm(n, 0, 1)

variance <- sum((data - mean(data))^2)/n

boots <- raply(1000, {
data_b <- sample(data, n, replace=T)
sum((data_b - mean(data_b))^2)/n
})

# estimated bias
mean(boots) - variance
#>  -0.06504726

# true bias:
((n-1)/n)*1 -1
#>  -0.05

We could instead take $t(F)$ to be the population mean and $s(x) = c$, situation where in most cases there should be a clear bias:

library(plyr)

mu <- 3
a_constant <- 1

n <- 20
data <- rnorm(n, mu, 1)

boots <- raply(1000, {
# not necessary as we will ignore the data, but let's do it on principle
data_b <- sample(data, n, replace=T)

a_constant
})

# estimated bias
mean(boots) - mean(data)
#>  -1.964877

# true bias is clearly -2

Again the bootstrap estimate isn't too bad.

• I've added this answer because the other answers seem to take for granted that it is a problem that the bootstrap estimate of bias is 0 when $t$ is a constant. I don't believe it is. – einar Oct 26 '17 at 17:08
• I like your answer and your demo, but I don't think your definition is correct "The bootstrap estimate of bias is an estimate of the bias between a function of your sample and the same function evaluated in the population." While what you write is well-defined, if this were the definition, there would be no way to use the bootstrap to estimate the bias of, e.g., the sample variance as an estimator for the population variance. – DavidR Apr 9 '18 at 22:48
• @DavidR You are right, thank you for commenting. I have updated the answer. – einar Apr 10 '18 at 8:33
• I like this writeup a lot! My only question is about "bootstrap estimate of bias". I think what you've written is the actual bias of the estimator (but for the empirical distribution rather than the true distribution), since you're taking an expectation over bootstrap samples. I think the bootstrap estimator would be a finite sum over B bootstrap samples? – DavidR Apr 10 '18 at 14:51
• @DavidR I'm glad you do! What I report is technically the bootstrap estimate of bias (because you use $t(\hat F)$ in place of $\theta$ and the bootstrap expectation of $s()$ in place of its expectation over $F$). But in most practical applications $E_{\hat F}[s(x^*)]$ is intractable and we approximate it by Monte Carlo as you say. – einar Apr 11 '18 at 7:19

I find it useful to think about the bootstrap procedures in terms of the functionals of the distributions they operate on -- I gave an example in this answer to a different bootstrap question.

The estimate you gave is what it is -- an estimate. Nobody says it does not suffer from problems that statistical estimates may have. It will give you a non-zero estimate of bias for the sample mean, for instance, which we all know is unbiased to begin with. One problem with this bias estimator is that it suffers from sampling variability when the bootstrap is implemented as Monte Carlo rather than a complete enumeration of all possible subsamples (and nobody that that theoretical bootstrap in practice, anyway).

As such, a Monte Carlo implementation of the bootstrap is unfixable, and you have to use a different bootstrap scheme. Davison et. al. (1986) demonstrated how to create a different bootstrap scheme that restricts the random draws to produce balanced samples: if you create $B$ bootstrap replicates, then each of the original elements needs to be used exactly $B$ times for the first-order balance. (The second order balance that works better for the second moments of the estimands, is further discussed by Graham et. al. (1990).)

• I think Bootstrapped's original question is orthogonal to the issue of Monte Carlo variability. Even if we take the number of bootstrap replications to infinity, the formula in the question will give a zero estimate for the bias of a constant estimator, and will give a nonzero estimate for the bias of the usual unbiased estimate of variance. – Evan Wright Dec 18 '14 at 17:02