I have some unusual data that I'm having trouble building a linear regression model for. I have a million observations, for about 300 groups, 3 predictors, and one outcome. Smallest group has about 30 observations, while larger groups have several thousand.
The problem is that the three predictors are highly collinear overall, but by group the trends change significantly. For one group, only one predictor might be present, for another group a different one might be present, the others would all be blank or irrelevant. In a third group, two of the predictors might have some meaningful information.
My initial approach was to build 300 linear models. For each model, I progmatically removed predictors that had a very high correlation with each other, or that were empty/missing, and then trained a simple linear regression on the remaining predictors. The models fit quite well.
The trouble is using 300 models to make predictions in production data. I'd like to stick with linear models, but not sure how best to proceed. Would a single mixed effects linear model be appropriate here, or should I consider models that relax the linear assumption and can learn group-wise relationships?
I'm still concerned by the fact that each group behaves very differently with regards to particular predictors.