Are mixed models useful as predictive models?

Question

I am a bit confused about advantages of mixed models in regard to predictive modelling. Since predictive models are usually meant to predict values of previously unknown observations then it seems obvious to me that the only way a mixed model may be useful is through its ability to provide population-level predictions (that is without adding any random effects). However, the problem is that so far in my experience population-level predictions based on mixed models are significantly worse than predictions based on standard regression models with fixed effects only.

So what is the point of mixed models in regard to prediction problems?

EDIT. The problem is the following: I fitted a mixed model (with both fixed and random effects) and standard linear model with fixed effects only. When I do cross-validation I get a following hierarchy of predictive accuracy: 1) mixed models when predicting using fixed and random effects (but this works of course only for observations with known levels of random effects variables, so this predictive approach seems not to be suitable for real predictive applications!); 2) standard linear model; 3) mixed model when using population-level predictions (so with random effects thrown out). Thus, the only difference between standard linear model and mixed model are somewhat different value of coefficients due to different estimation methods (i.e. there are the same effects/predictors in both models, but they have different associated coefficients).

So my confusion boils down to a question, why would I ever use a mixed model as a predictive model, since using mixed model to generate population-level predictions seems to be an inferior strategy in comparison to a standard linear model.

Answer 1

It depends on the nature of the data, but in general I would expect the mixed model to outperform the fixed-effects only models.

Let's take an example: modelling the relationship between sunshine and the height of wheat stalks. We have a number of measurements of individual stalks, but many of the stalks are measured at the same sites (which are similar in soil, water and other things that may affect height). Here are some possible models:

1) height ~ sunshine

2) height ~ sunshine + site

3) height ~ sunshine + (1|site)

We want to use these models to predict the height of new wheat stalks given some estimate of the sunshine they will experience. I'm going to ignore the parameter penalty you would pay for having many sites in a fixed-effects only model, and just consider the relative predictive power of the models.

The most relevant question here is whether these new data points you are trying to predict are from one of the sites you have measured; you say this is rare in the real world, but it does happen.

A) New data are from a site you have measured

If so, models #2 and #3 will outperform #1. They both use more relevant information (mean site effect) to make predictions.

B) New data are from an unmeasured site

I would still expect model #3 to outperform #1 and #2, for the following reasons.

(i) Model #3 vs #1:

Model #1 will produce estimates that are biased in favour of overrepresented sites. If you have similar numbers of points from each site and a reasonably representative sample of sites, you should get similar results from both.

(ii) Model #3 vs. #2:

Why would model #3 be better that model #2 in this case? Because random effects take advantage of shrinkage - the site effects will be 'shrunk' towards zero. In other words, you will tend to find less extreme values for site effects when it is specified as a random effect than when it is specified as a fixed effect. This is useful and improves your predictive ability when the population means can reasonably be thought of as being drawn from a normal distribution (see Stein's Paradox in Statistics). If the population means are not expected to follow a normal distribution, this might be a problem, but it's usually a very reasonable assumption and the method is robust to small deviations.

[Side note: by default, when fitting model #2, most software would use one of the sites as a reference and estimate coefficients for the other sites that represent their the deviation from the reference. So it may appear as though there is no way to calculate an overall 'population effect'. But you can calculate this by averaging across predictions for all of the individual sites, or more simply by changing the coding of the model so that coefficients are calculated for every site.]

Answer 2

Following up on mkt's excellent response: From my own personal experience developing predictive models in the health insurance field, incorporating random effects into predictive models (including machine learning models) has a number of advantages.

I'm often asked to build models predicting future claims outcomes for (e.g., future health expenses, length of stay, etc.) based on an individual's historical claims data. Frequently there are multiple claims per individual with correlated outcomes. Ignoring the fact that many claims are shared by the same patient would be throwing out valuable information in a predictive model.

One solution would be to create fixed effect indicator variables for each member in the dataset and use a penalized regression to shrink each of the member-level fixed effects separately. However, if there are thousands or millions of members in your data, a more efficient solution from both computational and predictive standpoints may be to represent the multiple member-level fixed effects as a single random effect term with a normal distribution.