51
$\begingroup$

Ok, this is a question that keeps me up at night.

Can the bootstrap procedure be interpreted as approximating some Bayesian procedure (except for the Bayesian bootstrap)?

I really like the Bayesian "interpretation" of statistics which I find nicely coherent and easy to understand. However, I also have a weakness for the bootstrap procedure which is so simple, yet delivers reasonable inferences in many situations. I would be more happy with bootstrapping, however, if I knew that the bootstrap was approximating a posterior distribution in some sense.

I know of the "Bayesian bootstrap" (Rubin, 1981), but from my perspective that version of the bootstrap is as problematic as the standard bootstrap. The problem is the really peculiar model assumption that you make, both when doing the classical and the Bayesian bootstrap, that is, the possible values of the distribution are only the values I've already seen. How can these strange model assumptions still yield the very reasonable inferences that bootstrap procedures yield? I have been looking for articles that have investigated the properties of the bootstrap (e.g. Weng, 1989) but I haven't found any clear explanation that I'm happy with.

References

Donald B. Rubin (1981). The Bayesian Bootstrap. Ann. Statist. Volume 9, Number 1 , 130-134.

Chung-Sing Weng (1989). On a Second-Order Asymptotic Property of the Bayesian Bootstrap Mean. The Annals of Statistics , Vol. 17, No. 2 , pp. 705-710.

$\endgroup$
2
  • 6
    $\begingroup$ I just wrote a blog post on "the bootstrap as a Bayesian model" (sumsar.net/blog/2015/04/…) that explores Bayesian "explanations" of the bootstrap. It doesn't directly answer the questions above but I hope it makes it clearer what the bootstrap is and what it does. $\endgroup$ Apr 22, 2015 at 7:56
  • $\begingroup$ Read muliere and secchi (1996) bayesian nonparametric predictive inference and bootstrap techniques. Thay address exactly your point! $\endgroup$
    – user98260
    Dec 15, 2015 at 11:12

3 Answers 3

40
$\begingroup$

Section 8.4 of The Elements of Statistical Learning by Hastie, Tibshirani, and Friedman is "Relationship Between the Bootstrap and Bayesian Inference." That might be just what you are looking for. I believe that this book is freely available through a Stanford website, although I don't have the link on hand.

Edit:

Here is a link to the book, which the authors have made freely available online:

http://www-stat.stanford.edu/~tibs/ElemStatLearn/

On page 272, the authors write:

In this sense, the bootstrap distribution represents an (approximate) nonparametric, noninformative posterior distribution for our parameter. But this bootstrap distribution is obtained painlessly — without having to formally specify a prior and without having to sample from the posterior distribution. Hence we might think of the bootstrap distribution as a “poor man’s” Bayes posterior. By perturbing the data, the bootstrap approximates the Bayesian effect of perturbing the parameters, and is typically much simpler to carry out.

One more piece of the puzzle is found in this cross validated question which mentions the Dvoretzky–Kiefer–Wolfowitz inequality that "shows [...] that the empirical distribution function converges uniformly to the true distribution function exponentially fast in probability."

So all in all the non-parametric bootstrap could be seen as an asymptotic method that produces "an (approximate) nonparametric, noninformative posterior distribution for our parameter" and where this approximation gets better "exponentially fast" as the number of samples increases.

$\endgroup$
6
  • 3
    $\begingroup$ While we always appreciate references to relevant material, this answer would be greatly improved if a brief summary of that section were included. $\endgroup$
    – cardinal
    Oct 3, 2013 at 19:54
  • 2
    $\begingroup$ The last bit from that section might be more useful: The bootstrap is an approximate non-parametric, non-informative posterior distribution for the estimated parameter. The whole section is worth a read. $\endgroup$
    – Fraijo
    Oct 3, 2013 at 20:40
  • 3
    $\begingroup$ Thanks for the link! If i read Hastie et al. right they show a correspondence between the non-parametric boostrap and the Bayesian bootstrap and claims that the former approximates the latter. They don't write much about why the bootstrap (bayesian or not) results in sensible inferences in the first place. What I was hoping for was something like: "Under [some general circumstances] the bootstrap approximates the true posterior distribution of the parameter/statistics with an error that is [something] and that depends on [this and that]". $\endgroup$ Oct 4, 2013 at 10:44
  • $\begingroup$ Thanks for the help in improving my answer. The clearest explanation I've heard for why the bootstrap works is that the sample that you just collected is the best representation that you have of the overall population. But I'm not enough of a probabilist to put that more formally. $\endgroup$
    – EdM
    Oct 4, 2013 at 17:30
  • 1
    $\begingroup$ Note that the section of Hastie et al. only shows approximate equivalence of the non-parametric bootstrap with the Bayesian posterior of a non-informative prior when estimating the parameters of a single multi-valued discrete variable from observed frequencies in a sample of data. As far as I understand, both the bootstrap and Bayesian inferences can be used to draw conclusions on the uncertainty of predictions of arbitrarily complex statistical models, in which case the approximate equivalence shown in Hastie et al. does not necessarily hold. $\endgroup$
    – ogrisel
    May 16, 2019 at 8:41
3
$\begingroup$

This is the latest paper I've seen on the subject:

@article{efr13bay,
author={Efron, Bradley},
title={Bayesian inference and the parametric bootstrap},
journal={Annals of Applied Statistics},
volume=6,
number=4,
pages={1971-1997},
year=2012,
doi={10.1214/12-AOAS571},
abstract={Summary: The parametric bootstrap can be used for the efficient
    computation of Bayes posterior distributions. Importance sampling formulas
    take on an easy form relating to the deviance in exponential families and
    are particularly simple starting from Jeffreys invariant prior. Because of
    the i.i.d. nature of bootstrap sampling, familiar formulas describe the
    computational accuracy of the Bayes estimates. Besides computational
    methods, the theory provides a connection between Bayesian and frequentist
    analysis. Efficient algorithms for the frequentist accuracy of Bayesian
    inferences are developed and demonstrated in a model selection example.},
keywords={Jeffreys prior; exponential families; deviance; generalized linear
    models},
classmath={*62F15 (Bayesian inference)
62F40 (Resampling methods)
62J12 (Generalized linear models)
65C60 (Computational problems in statistics)}}
$\endgroup$
3
  • 2
    $\begingroup$ My interpretation of the paper is that it describes a bootstrap method for calculating posterior distribution of a specified model, that is a method that can be used instead of e.g. metropolis sampling. I don't see that the paper discusses the connection between the non-parametric bootstrap model assumptions and Bayesian estimation... $\endgroup$ Oct 4, 2013 at 12:39
  • 1
    $\begingroup$ It does claim to do that. I haven't read the paper in detail. $\endgroup$ Oct 4, 2013 at 16:56
  • 6
    $\begingroup$ Frank: I did not get much out reading this paper by Efron - what he does can be seen as just a sequential importance sampler which starts from the likelihood and tries to get to the posterior (which often will work). Rubin's purpose in the 1981 paper was to question the appropriateness of bootstrap but Efron's apparently reached the opposite view. David Draper revisted it this summer in his JSM course and concluded its bad except when you get to see most of the possibilities in the sample. But see here normaldeviate.wordpress.com/2013/06/12/… $\endgroup$
    – phaneron
    Oct 9, 2013 at 19:15
1
$\begingroup$

I too was seduced by both bootstrapping and Bayes' theorem, but I couldn't make much sense of the justifications of bootstrapping until I looked at it from a Bayesian perspective. Then - as I explain below - the bootstrap distribution can be seen as a Bayesian posterior distribution, which makes the (a?) rationale behind bootstrapping obvious, and also had the advantage of clarifying the assumptions made. There is more detail of the argument below, and the assumptions made, in https://arxiv.org/abs/1803.06214 (pages 22-26).

As an example, which is set up on the spreadsheet at http://woodm.myweb.port.ac.uk/SL/resample.xlsx (click on the bootstrap tab at the bottom of the screen), suppose we've got a sample of 9 measurements with a mean of 60. When I used the spreadsheet to produce 1000 resamples with replacement from this sample and rounded the means off to the nearest even number, 82 of these means were 54. The idea of bootstrapping is that we use the sample as a "pretend" population to see how variable the means of samples of 9 are likely to be, so this suggests that the probability of a sample mean being 6 below the population mean (in this case the pretend population based on the sample with a mean of 60) is 8.2%. And we can come to a similar conclusion about the other bars in the resampling histogram.

Now let's imagine that the truth is that the mean of the real population is 66. If this is so our estimate of the probability of the sample mean being the 60 (i.e. the Data) is 8.2% (using the conclusion in the paragraph above remembering that 60 is 6 below the hypothesised population mean of 66). Let's write this as

P(Data given Mean=66) = 8.2%

and this probability corresponds to an x value of 54 on the resampling distribution. The same sort of argument applies to each possible population mean from 0, 2, 4 ... 100. In each case the probability comes from the resampling distribution - but this distribution is reflected about the mean of 60.

Now let's apply Bayes' theorem. The measurement in question can only take values between 0 and 100, so rounding off to the nearest even number the possibilities for the population mean are 0, 2, 4, 6, ....100. If we assume that the prior distribution is flat, each of these has a prior probability of 2% (to 1 dp), and Bayes' theorem tells us that

P(PopMean=66 given Data)= 8.2%*2%/P(Data)

where

P(Data) = P(PopMean=0 given Data)*2%+ P(PopMean=2 given Data)*2% + ... + P(PopMean=100 given Data)*2%

We can now cancel the 2% and remember that sum of the probabilities must be 1 since the probabilities are simply those from the resampling distribution. Which leaves us with the conclusion that

P(PopMean=66)=8.2%

Remembering that 8.2% is the probability from the resampling distribution corresponding to 54 (instead of 66), the posterior distribution is simply the resampling distribution reflected about the sample mean (60). Further, if the resampling distribution is symmetrical in the sense that asymmetries are random - as it is in this and many other cases, we can take the resample distribution as being identical to the posterior probability distribution.

This argument makes various assumptions, the main one being that the prior distribution is uniform. These are spelled out in more detail in the article cited above.

$\endgroup$
1
  • $\begingroup$ There is such a thing as a Bayesian bootstrap that was introduced by Rubin. But I don't think that is what you are referring to. The ordinary bootstrap as introduced by Efron is really a frequentist concept. $\endgroup$ Apr 15, 2018 at 17:03

Not the answer you're looking for? Browse other questions tagged or ask your own question.