# Bootstrap and numerical optimization of statistic

Often times the bootstrap is used with a statistic that can be analytically evaluated (both in the real and the resampled datasets), e.g. the mean.

But if the statistic can not be analytically obtained (e.g., if we are maximizing a likelihood through some numerical optimization process), each bootstrap resample will contain an "extra" degree of uncertainty, as we are not really obtaining the true statistic for each resample but an approximation to it.

Does the bootstrap account for this "extra" uncertainty?

Old (more complicated) formulation of the question.

Given my observed data set $$X=\{x_i, i=1..N\}$$ and a model/likelihood $$L(\theta, X)$$, I want to find the MLE, i.e. the: $$\theta^*$$ values that maximize $$L$$ given $$X$$. I also want to assign a measure of uncertainty to $$\theta^*$$, so I apply a bootstrap process and calculate the standard deviation of the parameters.

The issue is that I can not maximize $$L$$ analytically, I have to resort to numerical optimization. My set up currently is as follows:

1. Maximize $$L$$ using my observed data set $$X$$ with 1000 steps of a numerical optimizer function (e.g.: genetic algorithm). This gives me $$\theta^*$$.
2. Maximize $$L$$ for 100 or more resamples with replacement of $$X$$ (i.e.: apply the bootstrap) but now using 100 steps of my numerical optimizer.

The reason for using less steps in 2. is that it would be prohibitively expensive timewise to run the bootstrap using the same 1000 steps I used to obtain the $$\theta^*$$ values.

My question is: do I need to account somehow for this "lower quality" optimization used in the bootstrap process?

• @Gabriel thanks for explaining. The bootstrap and precision of an iterative solver are unrelated concepts in that case. To assess MCMC error, you can look for "burn-in", varying initial conditions, etc. see here astrostatistics.psu.edu/RLectures/diagnosticsMCMC.pdf RE: bootstrap, most people do it wrong by obtaining a bootstrap sample of the statistic, calculating the mean and doing $\pm 1.96 * SE(\theta^*)$. See this post: stats.stackexchange.com/questions/20701/… for why that's wrong and what to do instead – AdamO May 17 at 16:15
• @Gabriel it sounds like there is a lot of work ahead of you. Some thoughts: most iterative solvers (Newton Raphson, Fischer Scoring, BGFS, etc.) stop on a tolerance criterion (e.g. the step difference is less than $10^{-6}$). If the tol condition isn't met after a fixed # of iterations, the algorithm is said not to have converged. Also the parameter estimates from the bootstrap distribution estimate the sampling distribution of the test statistic under the alternative hypothesis so the standard deviation of that bootstrap sample is in fact a standard error. – AdamO May 17 at 17:46