Why would you predict from a mixed effect model without including random effects for the prediction?

Question

This is more of a conceptual question, but as I use R I will refer to the packages in R. If the aim is to fit a linear model for the purposes of prediction, and then make predictions where the random effects might not be available, is there any benefit to using a mixed effects model, or should a fixed effect model be used instead?

For example, if I have data on weight vs. height with some other information, and build the following model using lme4, where subject is a factor with $n$ levels ($n=no.samples$):

mod1 <- lmer(weight ~ height + age + (1|subject), data=df, REML=F)

Then I want to be able to predict weight from the model using new height and age data. Obviously the by-subject variance in the original data is captured in the model, but is it possible to use this information in the prediction? Let's say I have some new height and age data, and want to predict weight, I can do so as follows:

predict(mod1,newdata=newdf) # newdf columns for height, age, subject

This will use predict.merMod, and I can either include a column for (new) subjects in newdf, or set re.form =~0. In the first instance, it is not clear what the model does with the 'new' subject factors, and in the second instance, will the by-subject variance captured in the model simply be ignored (averaged over) for the prediction?

In either case it would seem to me that a fixed effect linear model might be more appropriate. Indeed, if my understanding is correct, then a fixed effect model should predict the same values as the mixed model, if the random effect is not used in the prediction. Should this be the case? In R it is not, for example:

mod1 <- lmer(weight ~ height + age + (1|subject), data=df, REML=F)

predict(mod1,newdata=newdf, re.form=~0) # newdf columns for height, age, subject

yields different results to:

mod2 <- lm(weight ~ height + age, data=df)

predict(mod2,newdata=newdf) # newdf columns for height, age

---

Answer 1

Simple thought experiment: You have measured weight and height of 5 infants after birth. And you measured it from the same babies again after two years. Meanwhile you measured weight and height of your baby daughter almost every week resulting in 100 value pairs for her. If you use a mixed effects model, there is no problem. If you use a fixed effects model you put undue weight on the measurements from your daughter, to a point where you would get almost the same model fit if you used only data from her. So, it's not only important for inference to model repeated measures or uncertainty structures correctly, but also for prediction. In general, you don't get the same predictions from a mixed effects model and from a fixed effects model (with violated assumptions).

and I can either include a column for (new) subjects in newdf

You can't predict for subjects which were not part of the original (training) data. Again a thought experiment: the new subject is obese. How could the model know that it is at the upper end of the random effects distribution?

will the by-subject variance captured in the model simply be ignored (averaged over) for the prediction

If I understand you correctly then yes. The model gives you an estimate of the expected value for the population (note that this estimate is still conditional on the original subjects).