# Bootstrap confidence and prediction intervals of mixed effect model predictions

Let's say I fitted a mixed effect model mem with the lme4 R library, and I would like to use the bootMer function to compute confidence and prediction intervals on some data that the model has seen during training. In this case I can include random effects (as well as fixed effects) to compute the confidence and prediction intervals of mem at, say, a 95% confidence level.

For prediction intervals, this https://www.wavedatalabs.com.au/posts/2023-02-06-prediction-intervals-for-linear-mixed-effects-models/ source suggests to define a predict function that conditions on all random effects

predfn <- function(.) { predict(., newdata=new, re.form=NULL) }


and then to use it in the bootMer call, where also re.form=NULL

boot <- lme4::bootMer(mem, FUN=predfn, nsim=250, re.form=NULL, type="parametric")


Question 1) Is the above correct? And what about confidence intervals? More specifically: what would be the appropriate choice of re.form and FUN arguments in bootMer to estimate confidence intervals?

Let's now instead assume that we would like to compute confidence and prediction intervals at 95% confidence level for mem predictions on new data (not seen during training). In this case, random effects can not be considered. For prediction intervals, the same source I linked above suggests to define a new function to resample responses from mem (rather than resampling deterministic model predictions)

sfun <- function(.) {
simulate(., newdata=new_data, re.form=NULL, allow.new.levels=TRUE)[[1]]
}


and then use it in the bootMer call

boot <- lme4::bootMer(mem, FUN=sfun, nsim=250, re.form=~0, type="parametric", seed=100)


Question 2): Why was re.form=NULL used in sfun? Also: what would be the appropriate choice of re.form and FUN arguments of bootMer in this case to estimate the confidence intervals on the new_data

Any help appreciated!

## 1 Answer

The re.form argument in mixed-effects model predictions, including those performed with bootMer, dictates how random effects are incorporated into the predictions or simulations:

• re.form=NULL: Includes all random effects in the predictions, mirroring the structure and variability of the original model. This setting is typical when you want the predictions to reflect the complete variability modeled by both fixed and random effects.

• re.form=NA or re.form=~0: These settings ignore all random effects, making predictions based solely on the fixed effects of the model. This approach is used when the interest lies in generalizing the fixed effects across different levels or groups beyond those specifically modeled as random effects.

Question 1) Is the above correct? And what about confidence intervals? More specifically: what would be the appropriate choice of re.form and FUN arguments in bootMer to estimate confidence intervals?

Prediction Intervals:
For prediction intervals, including all random effects in the predictions is important when the data is the same as the training data. Setting re.form = NULL achieves this by incorporating all random effects in the bootstrapping process, reflecting the total variability in the data, both from fixed and random effects.

predfn <- function(.) { predict(., newdata=new, re.form=NULL) }
boot <- lme4::bootMer(mem, FUN=predfn, nsim=250, re.form=NULL, type="parametric")


This approach ensures that the uncertainty related to both fixed and random effects is captured in the intervals, used for scenarios where the predictions needs to account for all sources of variability represented in the model.

Confidence Intervals:
When estimating confidence intervals, the focus shifts to fixed effects only. Therefore, you should use re.form = NA or equivalently re.form = ~0, which excludes all random effects from the predictions where we want to assess the stability or reliability of the fixed effects across different samples:

cifn <- function(.) { predict(., newdata=new, re.form=NA) }
boot_ci <- lme4::bootMer(mem, FUN=cifn, nsim=250, re.form=NA, type="parametric")


Here, using re.form=NA ensures that the confidence intervals are calculated only on the fixed effects, providing insights into how these effects generalize across different contexts or settings, independent of the specific random effects in the model.

Let's say I fitted a mixed effect model mem with the lme4 R library, and I would like to use the bootMer function to compute confidence and prediction intervals on some data that the model has seen during training. In this case I can include random effects (as well as fixed effects) to compute the confidence and prediction intervals of mem at, say, a 95% confidence level.

For prediction intervals, we define a predict function that conditions on all random effects

predfn <- function(.) { predict(., newdata=new, re.form=NULL) }


and then to use it in the bootMer call, where also re.form=NULL

boot <- lme4::bootMer(mem, FUN=predfn, nsim=250, re.form=NULL, type="parametric")


Let's now instead assume that we would like to compute confidence and prediction intervals at 95% confidence level for mem predictions on new data (not seen during training). In this case, random effects can not be considered. For prediction intervals, the same source I linked above suggests to define a new function to resample responses from mem (rather than resampling deterministic model predictions)

sfun <- function(.) {
simulate(., newdata=new_data, re.form=NULL, allow.new.levels=TRUE)[[1]]
}


and then use it in the bootMer call

boot <- lme4::bootMer(mem, FUN=sfun, nsim=250, re.form=~0, type="parametric", seed=100)


Question 2): Why was re.form=NULL used in sfun? Also: what would be the appropriate choice of re.form and FUN arguments of bootMer in this case to estimate the confidence intervals on new_data.

Prediction Intervals:
For new data, where random effects cannot be reliably estimated or are not applicable, setting re.form = NA or re.form = ~0 in the prediction function is necessary to focus predictions on fixed effects:

sfun <- function(.) {
simulate(., newdata=new_data, re.form=NA, allow.new.levels=TRUE)[[1]]
}
boot <- lme4::bootMer(mem, FUN=sfun, nsim=250, re.form=~0, type="parametric", seed=100)


This method ensures that the simulation and bootstrapping processes focus on the fixed effects, appropriate for scenarios where the new data do not share the same random effects structure as the training data.

Confidence Intervals:
Similarly, for confidence intervals on new data, the setting should exclude random effects:

cifn_new <- function(.) { predict(., newdata=new_data, re.form=NA) }
boot_ci_new <- lme4::bootMer(mem, FUN=cifn_new, nsim=250, re.form=NA, type="parametric")


This is consistent with focusing solely on the fixed effects' contributions, as appropriate for generalising findings to broader contexts beyond the sampled data.

• Hi, thank you for the thorough explanation. I tried to apply your suggestion to the sleepstudy data from the lme4 library. I have fitted a very simple mixed effect model with Subject as random effect and Days as fixed effect, mem <- lmer(Reaction ~ Days + (1 | Subject), data = sleepstudy). The confidence and prediction intervals for a new subject look fine, but the confidence interval for a subject that was seen in model fit are shifted with respect to model prediction and prediction intervals. I would gladly share a reproducible script anywhere if you would like to Commented Jul 3 at 17:07
• Hi Marco, thanks, I hope it helped ?! This seems like it would be better posted as a new question. That way other people are likely to see it, and possibly respond 😊 Commented Jul 4 at 8:35