1
$\begingroup$

Often times the bootstrap is used with a statistic that can be analytically evaluated (both in the real and the resampled datasets), e.g. the mean.

But if the statistic can not be analytically obtained (e.g., if we are maximizing a likelihood through some numerical optimization process), each bootstrap resample will contain an "extra" degree of uncertainty, as we are not really obtaining the true statistic for each resample but an approximation to it.

Does the bootstrap account for this "extra" uncertainty?




Old (more complicated) formulation of the question.

Given my observed data set $X=\{x_i, i=1..N\}$ and a model/likelihood $L(\theta, X)$, I want to find the MLE, i.e. the: $\theta^*$ values that maximize $L$ given $X$. I also want to assign a measure of uncertainty to $\theta^*$, so I apply a bootstrap process and calculate the standard deviation of the parameters.

The issue is that I can not maximize $L$ analytically, I have to resort to numerical optimization. My set up currently is as follows:

  1. Maximize $L$ using my observed data set $X$ with 1000 steps of a numerical optimizer function (e.g.: genetic algorithm). This gives me $\theta^*$.
  2. Maximize $L$ for 100 or more resamples with replacement of $X$ (i.e.: apply the bootstrap) but now using 100 steps of my numerical optimizer.

The reason for using less steps in 2. is that it would be prohibitively expensive timewise to run the bootstrap using the same 1000 steps I used to obtain the $\theta^*$ values.

My question is: do I need to account somehow for this "lower quality" optimization used in the bootstrap process?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I don't know what you mean that "each bootstrap sample will contain an extra degree of uncertainty" outside of your choice to alter the optimization routine for the bootstrapped samples. The bootstrap sample itself is considered a fixed dataset. It is like the original unbootstrapped dataset in the sense that the bootstrap represents what you might observe if you redo the experiment.

Your choice to reduce the number of steps in the optimization routine for the bootstrapped dataset will introduce MCMC error. Perhaps not an issue if the MCMC error is sufficiently small: the bootstrap is preferred not because it is efficient. Unless it is a complicated EM algorithm at each step, it's unusual to require 100 or more steps to maximize a likelihood within a very acceptable tolerance.

The MCMC error does not need to be "accounted for". You just need to assess its extent and be sure it's a negligible fraction of the estimator error. I would be more concerned that you are not using the appropriate bootstrap approach to performing inference and calculating confidence intervals (unless you are doing a studentized bootstrap). Precisely one of the strengths of the bootstrap is that it does not require the sampling distribution of the statistic to be normal. You should consider quantile based intervals, studentized, double bootstrap, or BCA.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for the answer Adam. By "each bootstrap sample will contain an extra degree of uncertainty" I mean that the statistic is (can) not obtained with the complete accuracy one would get by analytically calculating e.g. a mean. The resampled bootstrap dataset is obtained with replacement in the usual way, this is not an issue (as fat as I can see). About the MCMC error you say: "You just need to assess its extent", how would I do this in this case? An why are you that you concerned that I am using the appropriate bootstrap approach? $\endgroup$
    – Gabriel
    Commented May 17, 2019 at 16:05
  • 1
    $\begingroup$ @Gabriel thanks for explaining. The bootstrap and precision of an iterative solver are unrelated concepts in that case. To assess MCMC error, you can look for "burn-in", varying initial conditions, etc. see here astrostatistics.psu.edu/RLectures/diagnosticsMCMC.pdf RE: bootstrap, most people do it wrong by obtaining a bootstrap sample of the statistic, calculating the mean and doing $\pm 1.96 * SE(\theta^*)$. See this post: stats.stackexchange.com/questions/20701/… for why that's wrong and what to do instead $\endgroup$
    – AdamO
    Commented May 17, 2019 at 16:15
  • $\begingroup$ I think I'm a bit lost now: what do Markov Chains have to do with the bootstrap? RE: bootstrap, I usually inspect the entire bootstrap distribution and summarize it reporting: mean, median, mode, stddev, and 16th-84th percentiles of the bootstrap distribution. $\endgroup$
    – Gabriel
    Commented May 17, 2019 at 17:04
  • 1
    $\begingroup$ @Gabriel Markov Chains and MCMC are different things. MCMC is generally a random process that leads to an estimator, but perhaps I misspoke as you said your solver is iterative. In either case, whether 100 or 1,000 iterations provides a stable estimate can be inspected by looking at burn in. It's good you store the bootstrapped estimates for analysis. When you said you calculate the standard error, it was worth noting that this is the way most people inappropriately perform bootstrap inference/estimation. $\endgroup$
    – AdamO
    Commented May 17, 2019 at 17:17
  • 1
    $\begingroup$ @Gabriel it sounds like there is a lot of work ahead of you. Some thoughts: most iterative solvers (Newton Raphson, Fischer Scoring, BGFS, etc.) stop on a tolerance criterion (e.g. the step difference is less than $10^{-6}$). If the tol condition isn't met after a fixed # of iterations, the algorithm is said not to have converged. Also the parameter estimates from the bootstrap distribution estimate the sampling distribution of the test statistic under the alternative hypothesis so the standard deviation of that bootstrap sample is in fact a standard error. $\endgroup$
    – AdamO
    Commented May 17, 2019 at 17:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.