Return to Question

Is it true that the percentile bootstrap method should not be used? If so, what alternatives are there for when $F$ isn't necessarily known (i.e., not enough information is available to do a parametric bootstrap)?

Edit: Because clarification has been requested, the empirical bootstrap refers to the following procedure:

When forming a $100(1-\alpha)$% confidence interval for $\theta$ in the form $\hat{\theta} \pm c \cdot \text{se}$, where $\text{se}$ is the standard error, gather estimates of $\text{se}$ using bootstrapping and use $\text{se}_{\alpha/2}$, $\text{se}_{1-\alpha/2}$, with subscripts denoting percentiles of the bootstrap estimates.

Paraphrased from Section 11.2 of Computer Age Statistical Inference by Efron and Hastie (2016).

The MIT notes linked above do something similar to this, but not exactly: they compute $\delta_1 = (\hat{\theta}^{*}-\hat{\theta})_{\alpha/2}$ and $\delta_2 = (\hat{\theta}^{*}-\hat{\theta})_{1-\alpha/2}$ with $\hat{\theta}^{*}$ the bootstrapped estimates of $\theta$ and $\hat{\theta}$ the full-sample estimate of $\theta$, and the resulting estimated confidence interval would be $[\hat{\theta}-\delta_1, \hat{\theta} - \delta_2]$.

In essence, the main idea is this: empirical bootstrapping estimates an amount proportional to the difference between the point estimate and the actual parameter, i.e., $\hat{\theta}-\theta$, and uses this difference to come up with the lower and upper CI bounds.

The percentile bootstrap refers to the following:

To form a $100(1-\alpha)$% confidence interval for $\theta(\mathbf{x})$, let $\hat{\theta}(\mathbf{x})$ be a statistic for $\theta(\mathbf{x})$, resample to compute $\hat{\theta}(\mathbf{x}^{*})$ with $\mathbf{x}^{*}$ a resampling of the same size as $\mathbf{x}$, and use $[\hat{\theta}(\mathbf{x}^{*})_{\alpha/2}, \hat{\theta}(\mathbf{x}^{*})_{1-\alpha/2}]$ as the confidence interval for $\theta(\mathbf{x})$.

Paraphrased from Section 11.2 of Computer Age Statistical Inference by Efron and Hastie (2016).

In this situation, we use bootstrapping to compute estimates of the parameter of interest and take the percentiles of these estimates for the confidence interval.

Please let me know if anything I wrote above is unclear or incorrect.

Is it true that the percentile bootstrap method should not be used? If so, what alternatives are there for when $F$ isn't necessarily known (i.e., not enough information is available to do a parametric bootstrap)?

###Update

Because clarification has been requested, the "empirical bootstrap" from these MIT notes refers to the following procedure: they compute $\delta_1 = (\hat{\theta}^{*}-\hat{\theta})_{\alpha/2}$ and $\delta_2 = (\hat{\theta}^{*}-\hat{\theta})_{1-\alpha/2}$ with $\hat{\theta}^{*}$ the bootstrapped estimates of $\theta$ and $\hat{\theta}$ the full-sample estimate of $\theta$, and the resulting estimated confidence interval would be $[\hat{\theta}-\delta_2, \hat{\theta} - \delta_1]$.

In essence, the main idea is this: empirical bootstrapping estimates an amount proportional to the difference between the point estimate and the actual parameter, i.e., $\hat{\theta}-\theta$, and uses this difference to come up with the lower and upper CI bounds.

The "percentile bootstrap" refers to the following: use $[\hat{\theta}^*_{\alpha/2}, \hat{\theta}^*_{1-\alpha/2}]$ as the confidence interval for $\theta$. In this situation, we use bootstrapping to compute estimates of the parameter of interest and take the percentiles of these estimates for the confidence interval.

In the MIT OpenCourseWare notes for 18.05 Introduction to Probability and Statistics, Spring 2014 (currently available here), it states:

The bootstrap percentile method is appealing due to its simplicity. However it depends on the bootstrap distribution of $\bar{x}^{*}$ based on a particular sample being a good approximation to the true distribution of $\bar{x}$. Rice says of the percentile method, "Although this direct equation of quantiles of the bootstrap sampling distribution with confidence limits may seem initially appealing, it’s rationale is somewhat obscure."[2] In short, don’t use the bootstrap percentile method. Use the empirical bootstrap instead (we have explained both in the hopes that you won’t confuse the empirical bootstrap for the percentile bootstrap).

[2] John Rice, Mathematical Statistics and Data Analysis, 2nd edition, p. 272

After a bit of searching online, this is the only quote I've found which outright states that the percentile bootstrap should not be used.

What I recall reading from the text Principles and Theory for Data Mining and Machine Learning by Clarke et al. is that the main justification for bootstrapping is the fact that $$\dfrac{1}{n}\sum_{i=1}^{n}\hat{F}_n(x) \overset{p}{\to} F(x)$$ where $\hat{F}_n$ is the empirical CDF. (I don't recall details beyond this.)

Is it true that the percentile bootstrap method should not be used? If so, what alternatives are there for when $F$ isn't necessarily known (i.e., not enough information is available to do a parametric bootstrap)?

Edit: Because clarification has been requested, the empirical bootstrap refers to the following procedure:

When forming a $100(1-\alpha)$% confidence interval for $\theta$ in the form $\hat{\theta} \pm c \cdot \text{se}$, where $\text{se}$ is the standard error, gather estimates of $\text{se}$ using bootstrapping and use $\text{se}_{\alpha/2}$, $\text{se}_{1-\alpha/2}$, with subscripts denoting percentiles of the bootstrap estimates.

Paraphrased from Section 11.2 of Computer Age Statistical Inference by Efrom and Hastie (2016).

The MIT notes linked above do something similar to this, but not exactly: they compute $\delta_1 = (\hat{\theta}^{*}-\hat{\theta})_{\alpha/2}$ and $\delta_2 = (\hat{\theta}^{*}-\hat{\theta})_{1-\alpha/2}$ with $\hat{\theta}^{*}$ the bootstrapped estimates of $\theta$ and $\hat{\theta}$ the full-sample estimate of $\theta$, and the resulting estimated confidence interval would be $[\hat{\theta}-\delta_1, \hat{\theta} - \delta_2]$.

In essence, the main idea is this: empirical bootstrapping estimates an amount proportional to the difference between the point estimate and the actual parameter, i.e., $\hat{\theta}-\theta$, and uses this difference to come up with the lower and upper CI bounds.

The percentile bootstrap refers to the following:

To form a $100(1-\alpha)$% confidence interval for $\theta(\mathbf{x})$, let $\hat{\theta}(\mathbf{x})$ be a statistic for $\theta(\mathbf{x})$, resample to compute $\hat{\theta}(\mathbf{x}^{*})$ with $\mathbf{x}^{*}$ a resampling of the same size as $\mathbf{x}$, and use $[\hat{\theta}(\mathbf{x}^{*})_{\alpha/2}, \hat{\theta}(\mathbf{x}^{*})_{1-\alpha/2}]$ as the confidence interval for $\theta(\mathbf{x})$.

Paraphrased from Section 11.2 of Computer Age Statistical Inference by Efrom and Hastie (2016).

In this situation, we use bootstrapping to compute estimates of the parameter of interest and take the percentiles of these estimates for the confidence interval.

Please let me know if anything I wrote above is unclear or incorrect.

In the MIT OpenCourseWare notes for 18.05 Introduction to Probability and Statistics, Spring 2014 (currently available here), it states:

The bootstrap percentile method is appealing due to its simplicity. However it depends on the bootstrap distribution of $\bar{x}^{*}$ based on a particular sample being a good approximation to the true distribution of $\bar{x}$. Rice says of the percentile method, "Although this direct equation of quantiles of the bootstrap sampling distribution with confidence limits may seem initially appealing, it’s rationale is somewhat obscure."[2] In short, don’t use the bootstrap percentile method. Use the empirical bootstrap instead (we have explained both in the hopes that you won’t confuse the empirical bootstrap for the percentile bootstrap).

[2] John Rice, Mathematical Statistics and Data Analysis, 2nd edition, p. 272

After a bit of searching online, this is the only quote I've found which outright states that the percentile bootstrap should not be used.

What I recall reading from the text Principles and Theory for Data Mining and Machine Learning by Clarke et al. is that the main justification for bootstrapping is the fact that $$\dfrac{1}{n}\sum_{i=1}^{n}\hat{F}_n(x) \overset{p}{\to} F(x)$$ where $\hat{F}_n$ is the empirical CDF. (I don't recall details beyond this.)

Is it true that the percentile bootstrap method should not be used? If so, what alternatives are there for when $F$ isn't necessarily known (i.e., not enough information is available to do a parametric bootstrap)?

Edit: Because clarification has been requested, the empirical bootstrap refers to the following procedure:

When forming a $100(1-\alpha)$% confidence interval for $\theta$ in the form $\hat{\theta} \pm c \cdot \text{se}$, where $\text{se}$ is the standard error, gather estimates of $\text{se}$ using bootstrapping and use $\text{se}_{\alpha/2}$, $\text{se}_{1-\alpha/2}$, with subscripts denoting percentiles of the bootstrap estimates.

Paraphrased from Section 11.2 of Computer Age Statistical Inference by Efron and Hastie (2016).

The MIT notes linked above do something similar to this, but not exactly: they compute $\delta_1 = (\hat{\theta}^{*}-\hat{\theta})_{\alpha/2}$ and $\delta_2 = (\hat{\theta}^{*}-\hat{\theta})_{1-\alpha/2}$ with $\hat{\theta}^{*}$ the bootstrapped estimates of $\theta$ and $\hat{\theta}$ the full-sample estimate of $\theta$, and the resulting estimated confidence interval would be $[\hat{\theta}-\delta_1, \hat{\theta} - \delta_2]$.

In essence, the main idea is this: empirical bootstrapping estimates an amount proportional to the difference between the point estimate and the actual parameter, i.e., $\hat{\theta}-\theta$, and uses this difference to come up with the lower and upper CI bounds.

The percentile bootstrap refers to the following:

To form a $100(1-\alpha)$% confidence interval for $\theta(\mathbf{x})$, let $\hat{\theta}(\mathbf{x})$ be a statistic for $\theta(\mathbf{x})$, resample to compute $\hat{\theta}(\mathbf{x}^{*})$ with $\mathbf{x}^{*}$ a resampling of the same size as $\mathbf{x}$, and use $[\hat{\theta}(\mathbf{x}^{*})_{\alpha/2}, \hat{\theta}(\mathbf{x}^{*})_{1-\alpha/2}]$ as the confidence interval for $\theta(\mathbf{x})$.

Paraphrased from Section 11.2 of Computer Age Statistical Inference by Efron and Hastie (2016).

In this situation, we use bootstrapping to compute estimates of the parameter of interest and take the percentiles of these estimates for the confidence interval.

Please let me know if anything I wrote above is unclear or incorrect.

In the MIT OpenCourseWare notes for 18.05 Introduction to Probability and Statistics, Spring 2014 (currently available here), it states:

The bootstrap percentile method is appealing due to its simplicity. However it depends on the bootstrap distribution of $\bar{x}^{*}$ based on a particular sample being a good approximation to the true distribution of $\bar{x}$. Rice says of the percentile method, "Although this direct equation of quantiles of the bootstrap sampling distribution with confidence limits may seem initially appealing, it’s rationale is somewhat obscure."[2] In short, don’t use the bootstrap percentile method. Use the empirical bootstrap instead (we have explained both in the hopes that you won’t confuse the empirical bootstrap for the percentile bootstrap).

[2] John Rice, Mathematical Statistics and Data Analysis, 2nd edition, p. 272

After a bit of searching online, this is the only quote I've found which outright states that the percentile bootstrap should not be used.

What I recall reading from the text Principles and Theory for Data Mining and Machine Learning by Clarke et al. is that the main justification for bootstrapping is the fact that $$\dfrac{1}{n}\sum_{i=1}^{n}\hat{F}_n(x) \overset{p}{\to} F(x)$$ where $\hat{F}_n$ is the empirical CDF. (I don't recall details beyond this.)

Is it true that the percentile bootstrap method should not be used? If so, what alternatives are there for when $F$ isn't necessarily known (i.e., not enough information is available to do a parametric bootstrap)?

In the MIT OpenCourseWare notes for 18.05 Introduction to Probability and Statistics, Spring 2014 (currently available here), it states:

The bootstrap percentile method is appealing due to its simplicity. However it depends on the bootstrap distribution of $\bar{x}^{*}$ based on a particular sample being a good approximation to the true distribution of $\bar{x}$. Rice says of the percentile method, "Although this direct equation of quantiles of the bootstrap sampling distribution with confidence limits may seem initially appealing, it’s rationale is somewhat obscure."[2] In short, don’t use the bootstrap percentile method. Use the empirical bootstrap instead (we have explained both in the hopes that you won’t confuse the empirical bootstrap for the percentile bootstrap).

[2] John Rice, Mathematical Statistics and Data Analysis, 2nd edition, p. 272

After a bit of searching online, this is the only quote I've found which outright states that the percentile bootstrap should not be used.

What I recall reading from the text Principles and Theory for Data Mining and Machine Learning by Clarke et al. is that the main justification for bootstrapping is the fact that $$\dfrac{1}{n}\sum_{i=1}^{n}\hat{F}_n(x) \overset{p}{\to} F(x)$$ where $\hat{F}_n$ is the empirical CDF. (I don't recall details beyond this.)

Is it true that the percentile bootstrap method should not be used? If so, what alternatives are there for when $F$ isn't necessarily known (i.e., not enough information is available to do a parametric bootstrap)?

Edit: Because clarification has been requested, the empirical bootstrap refers to the following procedure:

When forming a $100(1-\alpha)$% confidence interval for $\theta$ in the form $\hat{\theta} \pm c \cdot \text{se}$, where $\text{se}$ is the standard error, gather estimates of $\text{se}$ using bootstrapping and use $\text{se}_{\alpha/2}$, $\text{se}_{1-\alpha/2}$, with subscripts denoting percentiles of the bootstrap estimates.

Paraphrased from Section 11.2 of Computer Age Statistical Inference by Efrom and Hastie (2016).

The MIT notes linked above do something similar to this, but not exactly: they compute $\delta_1 = (\hat{\theta}^{*}-\hat{\theta})_{\alpha/2}$ and $\delta_2 = (\hat{\theta}^{*}-\hat{\theta})_{1-\alpha/2}$ with $\hat{\theta}^{*}$ the bootstrapped estimates of $\theta$ and $\hat{\theta}$ the full-sample estimate of $\theta$, and the resulting estimated confidence interval would be $[\hat{\theta}-\delta_1, \hat{\theta} - \delta_2]$.

In essence, the main idea is this: empirical bootstrapping estimates an amount proportional to the difference between the point estimate and the actual parameter, i.e., $\hat{\theta}-\theta$, and uses this difference to come up with the lower and upper CI bounds.

The percentile bootstrap refers to the following:

To form a $100(1-\alpha)$% confidence interval for $\theta(\mathbf{x})$, let $\hat{\theta}(\mathbf{x})$ be a statistic for $\theta(\mathbf{x})$, resample to compute $\hat{\theta}(\mathbf{x}^{*})$ with $\mathbf{x}^{*}$ a resampling of the same size as $\mathbf{x}$, and use $[\hat{\theta}(\mathbf{x}^{*})_{\alpha/2}, \hat{\theta}(\mathbf{x}^{*})_{1-\alpha/2}]$ as the confidence interval for $\theta(\mathbf{x})$.

Paraphrased from Section 11.2 of Computer Age Statistical Inference by Efrom and Hastie (2016).

In this situation, we use bootstrapping to compute estimates of the parameter of interest and take the percentiles of these estimates for the confidence interval.

Please let me know if anything I wrote above is unclear or incorrect.