I have some unusual data that I'm having trouble building a linear regression model for. I have a million observations, for about 300 groups, 3 predictors, and one outcome. Smallest group has about 30 observations, while larger groups have several thousand.

The problem is that the three predictors are highly collinear overall, but by group the trends change significantly. For one group, only one predictor might be present, for another group a different one might be present, the others would all be blank or irrelevant. In a third group, two of the predictors might have some meaningful information.

My initial approach was to build 300 linear models. For each model, I progmatically removed predictors that had a very high correlation with each other, or that were empty/missing, and then trained a simple linear regression on the remaining predictors. The models fit quite well.

The trouble is using 300 models to make predictions in production data. I'd like to stick with linear models, but not sure how best to proceed. Would a single mixed effects linear model be appropriate here, or should I consider models that relax the linear assumption and can learn group-wise relationships?

I'm still concerned by the fact that each group behaves very differently with regards to particular predictors.


This seems like a straigtforward use case for a random coefficients model.

Basically: $$ y_{ig} = \alpha_i + \mathbf{X}_{ig}\beta + \displaystyle\sum_g\left(\mathbf{1}(G == g)\circ\mathbf{X}_{ig}\right)\Gamma_g + \epsilon_{ig} $$

where $i$ is an individual, $g$ is the group, $\alpha$ is a dummy-expanded factor variable, and $\mathbf{X}$ are your explanatory variables.

In other words, you've got an interaction between each variable and the group indicator.

If you fit this with OLS, you'll have 300 different regressions, in effect.

If you fit this with a random effects estimator (for example, lme4::lmer in R, you'll use something called partial pooling, which shrinks the group-specific coefficients towards their mean.

Interestingly, random effects models and penalized regression (L2 specifically) can be shown to be equivalent under some circumstances.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.