# Bootstrap instances of a statistic for MLE?

Let's say that I have real-valued random variables, $$\{X_1. \cdots, X_n \}$$, and some statistic $$T(X_1, \cdots, X_n)$$ for which I hypothesize might have a distribution $$f(T(X_1, \cdots, X_n); \vec{\theta})$$. Given a large simple random sample of instances of $$\{X_1. \cdots, X_n \}$$, I can estimate the statistic $$\hat{T}$$ using the entire sample.

I am considering the bootstrapping of this sample to produce multiple estimates of the statistic $$\hat{T}_{1}, \cdots, \hat{T}_k$$. Am I be able to effectively use the likelihood function $$\ell = \prod_{j}^{k} f(\hat{T}_j; \vec{\theta})$$ to find best-fitting values for $$\vec{\theta}$$?

Footnote: I mean aside from the considerations of the estimator itself, such as bias/consistency/sufficiency/completeness, is the bootstrapping approach for MLE valid?

Consider three confidence intervals for a sample of size $$n = 25$$ from the population $$\mathsf{Norm}(\mu=50, \sigma=7),$$ where $$\sigma=7$$ is known to us and $$\mu$$ is not. (Using R.)

set.seed(1114)
x = rnorm(25, 50, 7) # 7 known
summary(x); length(x);  sd(x)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
37.68   45.58   50.82   51.00   55.78   66.64
[1] 25
[1] 7.19531


The a 95% z confidence interval for unknown $$\mu$$ is of the form $$\bar X \pm 1.96\frac{7}{\sqrt{25}},$$ which computes to $$(48.25, 53.74).$$ Here we have used the MLE $$\bar X$$ of $$\mu.$$ [Notice that $$-1.96$$ is quantile 0.025 of the standard normal distribution and $$1.96$$ is its quantile $$.975.$$ Thus, the margin of error $$1.96\frac{7}{\sqrt{25}}$$ of this 95% z CI is $$2.73].$$

a = mean(x);  # MLE
[1] 50.99655
CI.z = a + qnorm(c(.025,.975))*7/5;  CI.z
[1] 48.2526 53.7405


Now we find a 95% parametric bootstrap CI for $$\mu.$$ Parametric means we know the population distribution is normal, and we have only to estimate the unknown parameter $$\mu.$$ We use the MLE as the estimator. I think this is the bootstrap CI in your Question.

At each of $$B = 5000$$ iterations, we estimate $$\mu$$ by $$\bar X$$ (a in the code), and take a 're-sample' of size $$n = 25$$ from the distribution $$\mathsf{Norm}(\bar X, 7)$$ "suggested" by the data x. Then we find the difference between the 're-sampled' mean $$\bar X$$ and the mean a from the original sample.

With many iterations this gives us an idea of the variability of $$\bar X$$ as an estimate of unknown $$\mu.$$ We take quantiles .025 and .975 of the resulting 'bootstrap distribution' of differences to make a 95% parametric bootstrap CI of $$\mu,$$ which is $$(48.26, 53.73),$$ very nearly the same as the z CI found above. It is no surprise that the two CIs agree because they are both based on knowing the population is normal and the MLE of $$\mu.$$ [Notice that the quantiles are very nearly $$\pm 1.96\frac{7} {\sqrt{25}} = \pm 2.73.$$ In effect, parametric bootstrapping has simulated the margin of error of the 95% z CI.]

set.seed(1234)
# param boot
a.obs = mean(x)
d.p = replicate(5000, mean(rnorm(25,a,7))-a.obs)
UL.p = quantile(d.p, c(.975,.025))
UL.p
97.5%      2.5%
2.738663 -2.730025

CI.p = a.obs - UL.p;  CI.p
97.5%     2.5%
48.25788 53.72657


To get a 95% nonparametric bootstrap CI for $$\mu,$$ we do not use the assumption that data are normal. We assume only that the sample was taken from some population distribution that has an unknown mean $$\mu.$$ We use the sample mean to estimate $$\mu$$ because we know that means of ever larger samples would converge to $$\mu.$$

Here, 're-sampling' is done by repeatedly taking samples of size $$n = 25$$ with replacement from the sample x. At each iteration we find the resampled $$\bar X^*$$ and its difference from the mean a of the original sample. Again, we use the bootstrap distribution of differences to make a 95% CI. The result is $$(48.70, 53.30).$$ This is a reasonable CI, but it is based on less assumed information.

set.seed(1235)
# nonpar boot
a.obs = mean(x)
d.n = replicate(2000, mean(sample(x,n,rep=T))-a.obs)
UL.n = quantile(d.n, c(.975, .025))
CI.n = a.obs - UL.n;  CI.n
97.5%     2.5%
48.69986 53.30470


Notes: (1) The nonparametric bootstrap uses only the sample of size 25 with minimum value $$37.68$$ and maximum $$66.64.$$ Because it does not assume normality it has 'no clue' that values out to $$\pm\infty$$ are technically possible, even though values beyond $$\mu \pm 3\sigma$$ are rare. The nonparametric bootstrap CI is a little shorter than the other two CIs shown earlier.

(2) I used more iterations $$B = 5000$$ for the parametric bootstrap than $$B = 2000$$ for the nonparametric bootstrap. Because the parametric bootstrap uses the normal distribution, experience has shown that additional iterations can give slightly better accuracy. However, for practical purposes, the nonparametric bootstrap, using only the 25 observations in x, does not ordinarily get 'better' with more than about $$2000$$ iterations.

(3) Bootstrapping gives interval estimates. It benefits from using 'good' estimators, and it assumes that the target quantities (mean, variance, etc.) exist. (Not all distributions have mean and variances.)

However, a nonparametric bootstrap procedure does not explicitly use information about the shape of the population distribution or whether the point estimates at each iteration are MLEs of the target quantities. (If you knew enough about a population distribution to find MLEs for parameters, you might know enough to derive an exact CI.)

Addendum: The midrange (average of max and min) of a normal sample is an unbiased estimator of the mean $$\mu.$$ However, it is far from the best; for samples of size $$n=25$$ from $$\mathsf{Norm}(50,7),$$ the SD of the midrange is about 2.55 compared with the SD $$\sigma/\sqrt{n} = 7/5 = 1.4$$ for the sample mean.

A 95% nonparametric bootstrap Ci for $$\mu$$ based on the sample midrange of x is $$(49.42, 54.89),$$ which is a little wider than the 95% nonparametric bootstrap CI above $$(48.70, 53.30)$$ based on the sample mean. [In R, range returns min and max, so one averages min and max to get midrange.]

set.seed(1234)
# param boot using midrange
mr.obs = mean(range(x)); mr.obs
[1] 52.16115
d.p2 = replicate(5000, mean(rnorm(25,mr.obs,7))-mr.obs)
UL.p2 = quantile(d.p2, c(.975,.025))
CI.p2 = mr.obs - UL.p2;  CI.p2
97.5%     2.5%
49.42249 54.89118


The nonparametric bootstrap of midranges 'works,' but for our sample x, bootstrapping means works better.