3
$\begingroup$
library(lme4)
library(merTools)
data(sleepstudy)
fm1 <- lmer(Reaction ~ Days + (Days|Subject), data=sleepstudy)
PI <- predictInterval(merMod = fm1, newdata = sleepstudy,
                  level = 0.95, n.sims = 1000,
                  stat = "median", type="linear.prediction",
                  include.resid.var = TRUE)

My question is: is there a rule of thumb of what the value of n.sims should be?

EDIT

My data contains 20686 rows (i.e. 20686 response variables) and 20 predictors. For such dataset, how many bootstrap samples are required? Is there any plots or papers that I can refer to that explain the number of bootstrap samples as a function of data size?

$\endgroup$
1
$\begingroup$

In addition to what can be found in the linked answer, some pointers:

  • More is better. Bootstrapping takes random subsamples with replacement, so the more often you do this, the smaller the chance your results are affected by the stochastic nature of this approach. For your final model you should do whatever number of bootstrap samples $B$ is still computationally feasible.

  • If your data set is very large, or your model very complex, then $B=1,000$ may be prohibitively large for simple trial and error, so you could start with a lower number, bearing in mind that the variance will be larger.

For example, if $B=1,000$ takes 10 minutes to run on your system, then simply halving to $B=500$ will already allow you to do twice as much trial and error in the same time. Your final comparisons could then be run overnight with a much larger number (e.g. $B=10,000$).

$\endgroup$
  • $\begingroup$ Okay. Thanks. My data contains 20686 rows (i.e. 20686 response variables) and 20 predictors. So what does your 2nd point mean in this context? $\endgroup$ – 89_Simple Oct 25 '18 at 9:09
  • $\begingroup$ Try running bootstrap with e.g. $10$ samples, not to draw any conclusions from it, but just to figure out how well your computer/server is handling that. Based on that you can decide how many bootstrap samples you want to use for initial trial and error. Once you know which models you want to compare, run them with a larger number of bootstrap samples. $\endgroup$ – Frans Rodenburg Oct 25 '18 at 9:12
  • $\begingroup$ @Crop89, to respond to your question edit, the number of bootstrap samples does not depend on your sample size, but it will mean that bootstrap is more computationally intensive (bootstrap samples are of the same size as your sample size). $\endgroup$ – Frans Rodenburg Oct 25 '18 at 9:14
  • 1
    $\begingroup$ okay thanks @Frans and also thanks for the linked answers. Quite a useful discussion. Regards $\endgroup$ – 89_Simple Oct 25 '18 at 9:19

Not the answer you're looking for? Browse other questions tagged or ask your own question.