Comparison between a multilevel and an unpooled model

Question

Suppose we have fitted two models: a multilevel model and an unpooled model:

   m1=lmer(y~x+(1|group))
   m2=lm(y~x+factor(group)-1)

How can I understand which is better? I would calculate AIC and BIC and select the model with the lowest score. Then I thought that I could check the sample size for each group in order to understand if it is enough to estimate a no polling model. If there are groups with a low sample size, I would select the multilevel model. Then I thought I could compare the standard errors of the regression coefficients by:

se.ranef(m1)
se.coef(m2) 

and select the model which produces more accurate estimates.
Is it correct?

Answer 1

First of all, you have different models you want to compare! In m2 you drop the intercept and this is generally a bad idea. If you use mixed model you have a random sample of groups, while in the case of linear regression you are interested in estimating the fixed effects of certain groups (or, you could say, the differences between groups) - so those are totally different models with different data structure assumed.

Using AIC and likelihood ratio tests should be used "with caution" for mixed models. Also in your code you have REML=TRUE (the default), so AIC of this model is based on restricted likelihood, while for the linear model based on likelihood. If you want to compare different mixed models based on likelihood-based measures you should switch to REML=FALSE (see here or here).

Why you assume small sample sizes to make the data more suitable for a mixed model? Generally, small sample sizes for groups are also not good for mixed models.

Another thing: you want to compare standard errors for linear regression vs. standard errors for random effects in linear mixed model - those are two different things. You compare errors for the whole model vs. errors for just a part of model. You write about comparing accuracy of predictions, but random effect estimates are rather predictions, not estimates. You use random effects when you have a random sample from the population and you care about the population effects, rather then individuals (like with linear regression you care about precise estimate on average rather then estimate that fits for every observation).

So no, this approach is generally not correct. Linear regression and linear mixed models are different models with different assumptions. You should choose one of those based on what is your data and what do you want to model. Read more on fixed and random effects to get a better feeling of the differences. Check also this book:

Gelman, A. & Hill, J. (2006). Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge University Press.

it gives a nice introduction to linear mixed models (and using lme4). You'll find there also a comparison between linear regression and mixed models.