I am trying to understand the connection between Bootstrap and Maximum Likelihood Estimator. Is the connection of the following format?

Let $ \mathcal{F}_{\theta} $ be a parametrized family of distributions. Let us consider a distribution $ \mathcal{F}_{\theta_0} $ with unknown value of $ \theta_0 $. Let $ \vec{X} $ be an iid sample of size N from distribution $ \mathcal{F}_{\theta_0} $. And let $ \theta^* $ the MLE estimate of $ \theta $ under the above assumptions, that is

\begin{align*} \theta^* = \arg\max_{\theta} \prod_{x \in \vec{X}} P_{\mathcal{F}_{\theta}}(x) \ . \end{align*}

We are concerned about the variance of a statistics $ S(\vec{X}) $ over the sampled data $ \vec{X} $. Let $ var_B S(\vec{X}) $ be a bootstrapped variance of statistics $ S $ computed over B rounds, that is, we sample a sample $\vec{X}_i$ with replacement from $ \vec{X} $ and compute variance of $ \{S(\vec{X}_i), i=1,\dots,B \} $.

Then \begin{align*} \lim _{B\to \infty} [var_B S(\vec{X})] = var_{x \sim \mathcal{F}_{\theta^*}} S(x) \ . \end{align*} Is the limit correct? Does N have to go to infinity as well?

(I would appreciate answers beyond simple references).

Edit: Note that in the right side of the equation, $x$ is sampled from $\mathcal{F}_{\theta^*}$, not $\mathcal{F}_{\theta_0}$.


1 Answer 1


Ok, so after the edit it's clearer that the question is different from what I thought. The answer is still 'No'.

Suppose $X$ has a Cauchy distribution and take $S(X)=\bar X$. $\mathrm{var}_B[\bar X]$ is always finite, and has a finite limit as $B\to\infty$ for fixed $N$, but $\mathrm{var}_{X\sim\textrm{Cauchy}(\theta^*)}[\bar X]$ is infinite.

Less extreme, take $X\sim N(\mu,1)$. Then $\mathrm{var}_{X\sim P_{\theta^*}}[\bar X]=1/N$, but $$\lim_{B\to\infty}\mathrm{var}_{B}[\bar X]=\hat\sigma^2/N$$ where $\hat\sigma^2$ is the sample variance in that sample.

In the second case the equality straightforwardly holds for the limit as $N\to\infty$. In the first case it holds in the sense that the bootstrap variance increases a.s. without bound as $N\to\infty$.

  • $\begingroup$ Thank you for your answer. But the MLE estimator for sample $\overrightarrow{X}$ is $\mu = \bar{X}$ where, I assume that, by $\bar{X}$ you meant an average among 0 and 1 values from the Bernoulli sampled $\overrightarrow{X}$. For your example, the original limit equation holds. Am I missing something? $\endgroup$
    – ashim
    Jul 12, 2021 at 1:24
  • 1
    $\begingroup$ A question on the 2nd example. Shouldn't the information that the variance equals 1 be used in the lim𝐵→∞var𝐵[𝑋¯] case as well? $\endgroup$
    – ashim
    Jul 12, 2021 at 22:05
  • $\begingroup$ Not if what you want is the bootstrap variance. You could set up a resampling estimator that made the same assumptions as the MLE, and then it would give the same answer as the MLE (eg, a parametric bootstrap), but an ordinary bootstrap doesn't $\endgroup$ Jul 13, 2021 at 0:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.