I am trying to use non-parametric bootstrap methods to get confidence intervals for a multi-step estimator that combines multiple variables:

$\hat\theta = \sqrt{\sum\limits_{i=1}^{n} {\left(\sum\limits_{t=1}^{m}|\overline{X_{i,t}} - \overline{Y_{i,t}}|\right)}^2}$

I find a bootstrapped distribution $\hat\theta^\ast_b$ by resampling each $X_{i,t}$ and $Y_{i,t}$ independently 10,000 times.

Since $|\overline{X_{i,t}} - \overline{Y_{i,t}}|$ is already a biased estimator, the bootstrap distributions at this stage (orange in the figure below) are already systematically positively biased relative to the estimated $|\overline{X_{i,t}} - \overline{Y_{i,t}}|$ (blue line in the figure below), especially when $|\overline{X_{i,t}} - \overline{Y_{i,t}}|$ is small:

enter image description here

The vector norm operation then compounds these biases. The result is that $\hat\theta$ (green line in the plot below) sometimes falls completely outside of the bootstrapped distribution $\hat\theta^\ast_b$ (red in the plot below).

enter image description here

I understand that BCa (bias corrected and accelerated) bootstrap is a preferred method for generating confidence intervals in the presence of bias and skew. However, this is problematic here, since $z_0 = -\inf$ when all $\hat\theta^\ast_b > \hat\theta$.

Thus my questions are:

Would it be valid to use bootstrap bias estimation to adjust for estimated bias by reporting $\tilde\theta$, where $\tilde\theta = 2\times{\hat\theta} - \bar{\hat\theta^\ast}$, and reporting percentile confidence intervals by re-centering $\hat\theta^\ast_b$ around $\tilde\theta$?

Should such bias correction be applied to each $|\overline{X_{i,t}} - \overline{Y_{i,t}}|$ before combining them?

If all I am interested in is relative values of $\hat\theta$ under different conditions, should I just report the means of the bootstrapped distributions, $\overline{\hat\theta^\ast}$ with percentile confidence intervals?

Is it even valid to use non-parametric bootstrap methods in this case?

Thanks very much for your consideration.

  • $\begingroup$ Have you tried the BCa bootstrap in any event? This problem seems very similar to the one on this page, for the intrinsically biased statistic of Shannon entropy. Sometimes you need to do some transformation on your statistic to get it closer to a pivotal quantity. Sometimes nothing works. See discussion and references on this page. Please look at those discussions, see if they help, and perhaps then edit this question to focus on any yet-unresolved problems. $\endgroup$
    – EdM
    Aug 20, 2020 at 23:55
  • $\begingroup$ Thanks for those links. I am unclear on how to implement the BCa in this case, since the $z_0$ parameter relies on the inverse normal of the number of bootstrapped estimates that are less than the empirical estimate, which in this case is 0, resulting in $z_0=-\inf$. It seems one solution from the Shannon Entropy discussion is to apply a bias adjustment term in the calculation of the entropy itself. I am trying to figure out if there is such an adjustment I can apply during the absolute difference step and the vector norm step in my formula. $\endgroup$ Aug 22, 2020 at 1:48

1 Answer 1


What you are plotting is the distribution of your statistic among the bootstrapped samples, the basis for the percentile bootstrap estimate of CI. Oddly enough, the percentile bootstrap method doesn't actually follow the bootstrap principle, which is that the relationship of your bootstrapped re-samples to your data sample represents the relationship of your data sample to the underlying population.

If you were following that principle, you'd be estimating the CI for the population by looking at the relationship between the values calculated from the bootstrapped samples and the value calculated on the full data sample. That's not what the percentile bootstrap does. As shown on this page, in this respect the percentile bootstrap is doubly biased in its mean values and, with a skewed distribution, reverses the direction of the skew.

The empirical or basic bootstrap, in contrast, follows the bootstrap principle. You examine the distribution of the differences of the statistic values among your bootstrapped samples from the value calculated on your data sample. That provides an estimate of the bias in your statistic, which you then use to correct both the mean value and the CI from your sample back to the estimated values in the population.

You can see that difference clearly on this page if you compare values of percentile and basic bootstrap CI for the necessarily biased plug-in estimate of the Shannon entropy. The entropy calculated on the data sample was 3.68, and bootstrapping gave an estimate of bias of -0.25. Thus the estimate for the population, corrected for bias, is 3.93. That's nicely within the CI calculated by the basic method (3.81, 4.05) but wildly beyond those calculated by the percentile method (3.31, 3.55), which didn't even include the original plug-in estimate of the entropy.

So I'd recommend starting with the empirical/basic bootstrap here. That provides the bias estimate needed to correct the statistic calculated on your data sample, and the CI, back to the underlying population. You have all the data you need for that, it's just based on the distribution of the differences of the values of the bootstrapped statistic values from the statistic calculated on the full data sample. As your distribution doesn't seem to have much skew, that should represent your situation pretty well, although I can't promise that your 95% CI will necessarily have 95% coverage. But that's always a problem with a non-pivotal statistic. See the references in this answer for further information.

  • $\begingroup$ Thanks so much for this comment. Somehow I had read those pages yet did not digest how the empirical bootstrap could adjust for bias and then set CIs around that adjusted value. I will go ahead with the empirical bootstrap. Can't thank you enough for helping me clear that up. $\endgroup$ Aug 26, 2020 at 12:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.