# Fitting same classes of many many collinear variables

I am trying to build a linear model of the simple form,

y = a + b1*x1 + b2*x2 + b3*x3 + c1*z1 + c2*z2 + c3*z3....

The characteristics of the model are

1) Each of the independent variable belongs to a particular class, so above you see, x1,x2,x3 are 3 variables that belong to class x. Same for the z variable.

2) There is a strong correlation between the variables in the same class, also variables that belong to different classes are correlated but less strongly so.

3) The number of variables are high and the number of observations are very high. For instance the number of variables could be a total of say 120 variables, and the observations or the y values could be say 1 million.

So this is a rather big multicollinear problem. To the experts out there, what would be the best way to attack such a problem. Essentially I want to be able to find out efficient ways to tackle multicollinearity. Possible ideas,

1) Use a regularization algorithm on the full blown problem.

2) Use a method to reduce the count of the variables that belong to the same class, for instance if x1 has strongest correlation with y, in its class, just discard x2 and x3. I would have uses pca here, but there is value in having a sparse solution, in other words I would rather have the entire weight on a single variable and discard the rest, as opposed to pca giving me an optimal vector, with some fractional weights on x1,x2 and x3

3) others?

Thanks.

• 120 variables is not a high number for a million observations. Multicollinearity will be a problem only when the standard errors of the estimates become too large to be useful--and the large number of observations will be a great help in reducing all the SEs. One might start, then, very simply by estimating the full model and inspecting the SEs: maybe they will show there isn't any problem at all to deal with.
– whuber
Jun 16, 2014 at 14:25