43
$\begingroup$

I am a bit confused about advantages of mixed models in regard to predictive modelling. Since predictive models are usually meant to predict values of previously unknown observations then it seems obvious to me that the only way a mixed model may be useful is through its ability to provide population-level predictions (that is without adding any random effects). However, the problem is that so far in my experience population-level predictions based on mixed models are significantly worse than predictions based on standard regression models with fixed effects only.

So what is the point of mixed models in regard to prediction problems?

EDIT. The problem is the following: I fitted a mixed model (with both fixed and random effects) and standard linear model with fixed effects only. When I do cross-validation I get a following hierarchy of predictive accuracy: 1) mixed models when predicting using fixed and random effects (but this works of course only for observations with known levels of random effects variables, so this predictive approach seems not to be suitable for real predictive applications!); 2) standard linear model; 3) mixed model when using population-level predictions (so with random effects thrown out). Thus, the only difference between standard linear model and mixed model are somewhat different value of coefficients due to different estimation methods (i.e. there are the same effects/predictors in both models, but they have different associated coefficients).

So my confusion boils down to a question, why would I ever use a mixed model as a predictive model, since using mixed model to generate population-level predictions seems to be an inferior strategy in comparison to a standard linear model.

$\endgroup$
13
  • $\begingroup$ How are you doing your predictions? Are you not using the random effects or are you fixing your random effects at their means? (I.e. are you throwing out the random effects at prediction time?) $\endgroup$
    – Wayne
    Dec 7, 2016 at 22:12
  • $\begingroup$ As far as I understand random effects correctly, fixing random effects at their means is the same as throwing them out, since random effects (at least in the parametrization I use) are generated from a normal distributions of means 0 and variance sigma. But anyways, since I do not know the values of random effects variables for new observations then I of course do not use random effects at prediction time, only fixed effects. $\endgroup$
    – sztal
    Dec 7, 2016 at 22:16
  • 1
    $\begingroup$ You may want to look through this paper, "On the Effectiveness of Mixed Model Based Logistic Regression Classifiers for Longitudinal Data", search.proquest.com/openview/3578d64c85f3c1c52414924d044bca2c/… $\endgroup$
    – Jon
    Dec 7, 2016 at 22:27
  • 1
    $\begingroup$ sztal: You're right of course. I was trying to respond quickly and said something not meaningful. I did find the paper (gllamm.org/JRSSApredict_09.pdf) which addresses prediction in Section 7. I have to say I can't summarize it down to a comment, which indicates I don't truly understand it. $\endgroup$
    – Wayne
    Dec 7, 2016 at 23:06
  • $\begingroup$ One last question: when you compare fixed-effects-only to mixed-effects, are you using the same fixed effects in each, with only the addition of something like an individual-level intercept? It feels like in this situation, you should have very similar effects, except with a better idea of your true predictive interval. $\endgroup$
    – Wayne
    Dec 7, 2016 at 23:10

2 Answers 2

33
$\begingroup$

It depends on the nature of the data, but in general I would expect the mixed model to outperform the fixed-effects only models.

Let's take an example: modelling the relationship between sunshine and the height of wheat stalks. We have a number of measurements of individual stalks, but many of the stalks are measured at the same sites (which are similar in soil, water and other things that may affect height). Here are some possible models:

1) height ~ sunshine

2) height ~ sunshine + site

3) height ~ sunshine + (1|site)

We want to use these models to predict the height of new wheat stalks given some estimate of the sunshine they will experience. I'm going to ignore the parameter penalty you would pay for having many sites in a fixed-effects only model, and just consider the relative predictive power of the models.

The most relevant question here is whether these new data points you are trying to predict are from one of the sites you have measured; you say this is rare in the real world, but it does happen.

A) New data are from a site you have measured

If so, models #2 and #3 will outperform #1. They both use more relevant information (mean site effect) to make predictions.

B) New data are from an unmeasured site

I would still expect model #3 to outperform #1 and #2, for the following reasons.

(i) Model #3 vs #1:

Model #1 will produce estimates that are biased in favour of overrepresented sites. If you have similar numbers of points from each site and a reasonably representative sample of sites, you should get similar results from both.

(ii) Model #3 vs. #2:

Why would model #3 be better that model #2 in this case? Because random effects take advantage of shrinkage - the site effects will be 'shrunk' towards zero. In other words, you will tend to find less extreme values for site effects when it is specified as a random effect than when it is specified as a fixed effect. This is useful and improves your predictive ability when the population means can reasonably be thought of as being drawn from a normal distribution (see Stein's Paradox in Statistics). If the population means are not expected to follow a normal distribution, this might be a problem, but it's usually a very reasonable assumption and the method is robust to small deviations.

[Side note: by default, when fitting model #2, most software would use one of the sites as a reference and estimate coefficients for the other sites that represent their the deviation from the reference. So it may appear as though there is no way to calculate an overall 'population effect'. But you can calculate this by averaging across predictions for all of the individual sites, or more simply by changing the coding of the model so that coefficients are calculated for every site.]

$\endgroup$
2
  • $\begingroup$ Thanks for the answer. I am rather convinced. Unfortunately I do not remember now the exact case that motivated my question, but I think that poor performance of a mixed model in my case could have been due to quite irregular distributions of predictors I used in the model. I will soon accept the answer, but since the question attracted a bit of attention I will give a few more days so someone maybe can deliver a more strict explantion (perhaps with some examples). $\endgroup$
    – sztal
    Jul 10, 2017 at 10:07
  • 1
    $\begingroup$ Well put. It should be noted that heirarchically predicting the site-specific effects that give rise to the random effects, to predict whatever random intercept or random slope was empirically estimated from the model should be equivalent to just having a bunch of individual level and site specific fixed effects in the model. $\endgroup$
    – AdamO
    Jul 22, 2019 at 14:26
16
$\begingroup$

Following up on mkt's excellent response: From my own personal experience developing predictive models in the health insurance field, incorporating random effects into predictive models (including machine learning models) has a number of advantages.

I'm often asked to build models predicting future claims outcomes for (e.g., future health expenses, length of stay, etc.) based on an individual's historical claims data. Frequently there are multiple claims per individual with correlated outcomes. Ignoring the fact that many claims are shared by the same patient would be throwing out valuable information in a predictive model.

One solution would be to create fixed effect indicator variables for each member in the dataset and use a penalized regression to shrink each of the member-level fixed effects separately. However, if there are thousands or millions of members in your data, a more efficient solution from both computational and predictive standpoints may be to represent the multiple member-level fixed effects as a single random effect term with a normal distribution.

$\endgroup$
2
  • $\begingroup$ Any readings on this issue? I am working in a project related with health data in Brazil. I have a binary outcome (40% of 1, 60% of 0). When I run a empty or "null" random intercept model, with hospitals as second level, I get 94% of accuracy in prediction of the outcome. If I add independent variables, I get significant coefficients but the model fit does not get improved.... $\endgroup$
    – LuizZ
    Nov 14, 2020 at 0:53
  • 1
    $\begingroup$ @LuizZ I don't know if there are articles on this subject, however this post here in Cross Validated offers solutions using empirical Bayesian estimates and packages in R and Python: stats.stackexchange.com/questions/221358/… $\endgroup$
    – RobertF
    Nov 15, 2020 at 15:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.