The nice thing about bootstrap is it's (mostly) quite intuitive. So the question of whether to set resample the random effects really depends on whether you would view that future cases also would resample the random effects.
Perhaps the most traditional definition of a random factor is one where the categories of our random factor are just a randomly selected subset from a broader population of possibilities and that our interest is in generalising to that broader population. For example, when the random factor models the participants in your study: you are probably not that interested in these participants, so much as in people in general. A rerun of the experiment would resample the participants, and your bootstrap should too. To resample, we set
On the other hand, say we have a categorical predictor with a fixed set of categories that will always stay the same. We might model that as a fixed factor. But in some cases, we might enter it as a random factor, modelling all the categories as one big bell-curve blob of possibilities and drawing on the advantages of random effects, such as shrinkage. This can be particularly advantageous if there are a lot of categories, or some categories have only a few datapoints. Since the categories are fixed and will never change, they should not change in the bootstrap either. Do not resample, set
These descriptions sound like the traditional definitions of a random and fixed effects, respectively. So, it's interesting to see that the default standard errors output by fixed and mixed models (at least, on a simple example) output roughly the same numbers as the two
use.u possibilities. The code at the bottom produces this output (each is the standard error of the intercept from a different approach):
use.u.TRUE use.u.FALSE fixed.model mixed.model
0.06300992 0.34607292 0.06618366 0.34576152
One interesting side-note from this is that the default way we generate inferences on random effects may give extremely conservative inferences in case 2 above.
For cases where we have a mixture of types 1 and 2 above, it might be nice to be able to switch
use.u differently per random effect.
It's really important not to set
use.u=TRUE unless you're sure that's what you want. As you noted, the SEs (and CIs) can be wildly different and that's because the question you are asking is completely different.
For example, in an analysis I just ran, a number of replicates were taken from each beaker of "stuff". I model the beakers as a random factor. The replicates from each beaker all come out roughly the same, so the residual standard deviation is actually tiny: most of the sample-to-sample variation is encoded in the beaker random effect. If we don't model that variation beaker to beaker, you would think there was almost no noise! And consequently, your inferences would be WAY too optimistic, based on the assumption that things always come out the same.
n.per.group <- c(100,100)
nsim <- 500
num.groups <- length(n.per.group)
u <- rnorm(num.groups)
data <- data.frame(
y = unlist(mapply(rnorm, n=n.per.group, mean=u, SIMPLIFY=FALSE)),
group.num = unlist(mapply(rep, times=n.per.group, x=1:num.groups, SIMPLIFY=FALSE))
data$group.num <- factor(data$group.num)
# This sets up the fixed model so that the intercept models the grand mean
# (i.e. same as in the mixed model)
contrasts(data$group.num) <- contr.sum(levels(data$group.num))
fixed.model <- lm( y~ group.num, data=data)
mixed.model <- lmer(y~(1|group.num), data=data)
standard.errors <- c(
use.u.TRUE = sd(bootMer(mixed.model, fixef, use.u=TRUE, nsim=nsim)$t),
use.u.FALSE = sd(bootMer(mixed.model, fixef, use.u=FALSE, nsim=nsim)$t),
fixed.model = summary(fixed.model)$coefficients["(Intercept)", "Std. Error"],
mixed.model = summary(mixed.model)$coefficients["(Intercept)", "Std. Error"]