# Connection between Bootstrap and Maximum Likelihood Estimator

I am trying to understand the connection between Bootstrap and Maximum Likelihood Estimator. Is the connection of the following format?

Let $$\mathcal{F}_{\theta}$$ be a parametrized family of distributions. Let us consider a distribution $$\mathcal{F}_{\theta_0}$$ with unknown value of $$\theta_0$$. Let $$\vec{X}$$ be an iid sample of size N from distribution $$\mathcal{F}_{\theta_0}$$. And let $$\theta^*$$ the MLE estimate of $$\theta$$ under the above assumptions, that is

\begin{align*} \theta^* = \arg\max_{\theta} \prod_{x \in \vec{X}} P_{\mathcal{F}_{\theta}}(x) \ . \end{align*}

We are concerned about the variance of a statistics $$S(\vec{X})$$ over the sampled data $$\vec{X}$$. Let $$var_B S(\vec{X})$$ be a bootstrapped variance of statistics $$S$$ computed over B rounds, that is, we sample a sample $$\vec{X}_i$$ with replacement from $$\vec{X}$$ and compute variance of $$\{S(\vec{X}_i), i=1,\dots,B \}$$.

Then \begin{align*} \lim _{B\to \infty} [var_B S(\vec{X})] = var_{x \sim \mathcal{F}_{\theta^*}} S(x) \ . \end{align*} Is the limit correct? Does N have to go to infinity as well?

(I would appreciate answers beyond simple references).

Edit: Note that in the right side of the equation, $$x$$ is sampled from $$\mathcal{F}_{\theta^*}$$, not $$\mathcal{F}_{\theta_0}$$.

Suppose $$X$$ has a Cauchy distribution and take $$S(X)=\bar X$$. $$\mathrm{var}_B[\bar X]$$ is always finite, and has a finite limit as $$B\to\infty$$ for fixed $$N$$, but $$\mathrm{var}_{X\sim\textrm{Cauchy}(\theta^*)}[\bar X]$$ is infinite.
Less extreme, take $$X\sim N(\mu,1)$$. Then $$\mathrm{var}_{X\sim P_{\theta^*}}[\bar X]=1/N$$, but $$\lim_{B\to\infty}\mathrm{var}_{B}[\bar X]=\hat\sigma^2/N$$ where $$\hat\sigma^2$$ is the sample variance in that sample.
In the second case the equality straightforwardly holds for the limit as $$N\to\infty$$. In the first case it holds in the sense that the bootstrap variance increases a.s. without bound as $$N\to\infty$$.
• Thank you for your answer. But the MLE estimator for sample $\overrightarrow{X}$ is $\mu = \bar{X}$ where, I assume that, by $\bar{X}$ you meant an average among 0 and 1 values from the Bernoulli sampled $\overrightarrow{X}$. For your example, the original limit equation holds. Am I missing something? Jul 12, 2021 at 1:24