I am trying to build a linear model of the simple form,

y = a + b1*x1 + b2*x2 + b3*x3 + c1*z1 + c2*z2 + c3*z3....

The characteristics of the model are

1) Each of the independent variable belongs to a particular class, so above you see, x1,x2,x3 are 3 variables that belong to class x. Same for the z variable.

2) There is a strong correlation between the variables in the same class, also variables that belong to different classes are correlated but less strongly so.

3) The number of variables are high and the number of observations are very high. For instance the number of variables could be a total of say 120 variables, and the observations or the y values could be say 1 million.

So this is a rather big multicollinear problem. To the experts out there, what would be the best way to attack such a problem. Essentially I want to be able to find out efficient ways to tackle multicollinearity. Possible ideas,

1) Use a regularization algorithm on the full blown problem.

2) Use a method to reduce the count of the variables that belong to the same class, for instance if x1 has strongest correlation with y, in its class, just discard x2 and x3. I would have uses pca here, but there is value in having a sparse solution, in other words I would rather have the entire weight on a single variable and discard the rest, as opposed to pca giving me an optimal vector, with some fractional weights on x1,x2 and x3

3) others?


  • 1
    $\begingroup$ 120 variables is not a high number for a million observations. Multicollinearity will be a problem only when the standard errors of the estimates become too large to be useful--and the large number of observations will be a great help in reducing all the SEs. One might start, then, very simply by estimating the full model and inspecting the SEs: maybe they will show there isn't any problem at all to deal with. $\endgroup$
    – whuber
    Jun 16, 2014 at 14:25

1 Answer 1


My guess of your problem would be that x1 x2 x3 are just different measures of the same underlying (latent) concept. For example, x1 x2 and x3 are whether or not they solved 3 different problems correctly and the underlying concept is "problem solving ability" or "intelligence".

In that case I would use a SEM, such that you use measurement models to estimate the underlying latent variables and the relationship between these latent variables.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.