I'm having a trouble understanding what a Bayesian Bootstrapping process is, and how that would differ from your normal bootstrapping. And if someone could offer an intuitive/conceptual review and comparison of both, that would be great.

Let's take an example.

Say we have a dataset X that is [1,2,5,7,3].

If we sample with replacement multiple times to create sample sizes equal to the size of X (so, [7,7,2,5,7], [3,5,2,2,7], etc), and then we compute the means of each, is that the bootstrap distribution of the sample mean?

What would be the bayesian bootstrap distribution of that?

And how is the bayesian bootstrap distribution of other parameters (variance, etc) done in the same ways?


1 Answer 1


The (frequentist) bootstrap takes the data as a reasonable approximation to the unknown population distribution. Therefore, the sampling distribution of a statistic (a function of the data) can be approximated by repeatedly resampling the observations with replacement and computing the statistic for each sample.

Let $y = (y_1,\ldots,y_n)$ denote the original data (In the example given, $n=5$). Let $y^b = (y_1^b, \ldots, y_n^b)$ denote a bootstrap sample. Such a sample will likely have some observations repeated one or more times and other observations will be absent. The mean of the bootstrap sample is given by $$m_b = \frac{1}{n} \sum_{i=1}^n y_i^b.$$ It is the distribution of $m_b$ over a number of bootstrap replications that is used to approximate the sampling distribution from the unknown population.

In order to understand the connection between the frequentist bootstrap and the Bayesian bootstrap, it is instructive to see how to compute $m_b$ from a different perspective.

In each bootstrap sample $y^b$, each observation $y_i$ occurs anywhere from 0 to $n$ times. Let $h_i^b$ denote the number of times $y_i$ occurs in $y^b$, and let $h^b = (h_1^b, \ldots, h_n^b)$. Thus $h_i^b \in \{0, 1, \ldots, n-1,n\}$ and $\sum_{i=1}^n h_i^b = n$. Given $h^b$, we can construct a collection of nonnegative weights that sum to one: $w^b = h^b/n$, where $w_i^b = h_i^b/n$. With this notation we can reexpress the mean of the bootstrap sample as $$ m_b = \sum_{i=1}^n w_i^b\, y_i. $$

The way in which the observations are chosen for a bootstrap sample determines the joint distribution for $w^b$. In particular, $h^b$ has a multinomial distribution and thus $$(n\,w^b) \sim \textsf{Multinomial}(n,(1/n)_{i=1}^n).$$ Therefore, we can compute $m_b$ by drawing $w^b$ from its distribution and computing the dot product with $y$. From this new perspective, it appears that the observations are fixed while the weights are varying.

In Bayesian inference, the observations are indeed taken as fixed, so this new perspective seems congenial to the Bayesian approach. Indeed, the calculation of the mean according to the Bayesian bootstrap differs only in the distribution of the weights. (Nevertheless, from a conceptual standpoint the Bayesian bootstrap is quite different from the frequentist version.) The data $y$ are fixed and the weights $w$ are the unknown parameters. We may be interested in some functional of the data that depends on the unknown parameters: $$ \mu = \sum_{i=1}^n w_i\, y_i. $$

Here is a thumbnail sketch of the model behind the Bayesian bootstrap: The sampling distribution for the observations is multinomial and the prior for the weights is a limiting Dirichlet distribution that puts all its weight on the vertices of the simplex. (Some authors refer to this model as the multinomial likelihood model.)

This model produces the following posterior distribution for the weights: $$ w \sim \textsf{Dirichlet}(1,\ldots,1). $$ (This distribution is flat over the simplex.) The two distributions for the weights (frequentist and Bayesian) are quite similar: They have the same means and similar covariances. The Dirichlet distribution is 'smoother' than the multinomial distribution, so the Bayesian bootstrap may be call the smoothed bootstrap. We may interpret the frequentist bootstrap as an approximation to the Bayesian bootstrap.

Given the posterior distribution for the weights, we can approximate the posterior distribution of the functional $\mu$ by repeated sampling $w$ from its Dirichlet distribution and computing the dot product with $y$.

We can adopt the framework of estimating equations $$ \sum_{i=1}^n w_i\, g(y_i,\theta) = \underline 0, $$ where $g(y_i,\theta)$ is a vector of estimating functions that depends on the unknown parameter (vector) $\theta$ and $\underline 0$ is a vector of zeros. If this system of equations has a unique solution for $\theta$ given $y$ and $w$, then we can compute its posterior distribution by drawing $w$ from its posterior distribution and evaluating that solution. (The framework of estimating equations is used with empirical likelihood and with generalized method of moments (GMM).)

The simplest case is the one we have already dealt with: $$ \sum_{i=1}^n w_i\,(y_i - \mu) = 0. $$ For the mean and the variance, $\theta = (\mu,v)$ we have $$ g(y_i,\theta) = \begin{pmatrix} y_i - \mu \\ (y_i - \mu)^2 - v \end{pmatrix}. $$ The setup is a bit more involved than that for the frequentist bootstrap, which is why a Bayesian might adopt the frequentist bootstrap as a quick approximation.

  • 1
    $\begingroup$ Thanks for the very detailed description. Personally I would appreciate a brief statement on when to choose each one. $\endgroup$ Feb 18, 2019 at 20:27
  • 1
    $\begingroup$ Isn't a flat posterior an odd choice? I would have expected a flat distribution as prior rather than posterior. I couldn't find any discussions on this. Do you have any comments? $\endgroup$
    – Blade
    Aug 13, 2020 at 13:51
  • $\begingroup$ @Blade - I think the flat posterior is valid because all observed data points are equally likely to be chosen. I'm still trying to get my head around it myself, but this may help out: sumsar.net/blog/2015/04/… $\endgroup$
    – Matti Wens
    Feb 13, 2021 at 12:49
  • $\begingroup$ @MattWenham So the choice of prior is odd, and this is noted by Rubin himself. The choice of prior is set so that the posterior mimics the classic bootstrap. It's not that it's not valid, it's just that it is not much of a posterior when it's a flat distribution. You would expect a posterior give you some information based on some observation, but here what we have is the assumption that all distinctive values in the dataset have been observed. $\endgroup$
    – Blade
    Feb 14, 2021 at 2:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.