Fitting same classes of many many collinear variables

Question

I am trying to build a linear model of the simple form,

y = a + b1*x1 + b2*x2 + b3*x3 + c1*z1 + c2*z2 + c3*z3....

The characteristics of the model are

1) Each of the independent variable belongs to a particular class, so above you see, x1,x2,x3 are 3 variables that belong to class x. Same for the z variable.

2) There is a strong correlation between the variables in the same class, also variables that belong to different classes are correlated but less strongly so.

3) The number of variables are high and the number of observations are very high. For instance the number of variables could be a total of say 120 variables, and the observations or the y values could be say 1 million.

So this is a rather big multicollinear problem. To the experts out there, what would be the best way to attack such a problem. Essentially I want to be able to find out efficient ways to tackle multicollinearity. Possible ideas,

1) Use a regularization algorithm on the full blown problem.

2) Use a method to reduce the count of the variables that belong to the same class, for instance if x1 has strongest correlation with y, in its class, just discard x2 and x3. I would have uses pca here, but there is value in having a sparse solution, in other words I would rather have the entire weight on a single variable and discard the rest, as opposed to pca giving me an optimal vector, with some fractional weights on x1,x2 and x3

3) others?

Thanks.

Answer 1

My guess of your problem would be that x1 x2 x3 are just different measures of the same underlying (latent) concept. For example, x1 x2 and x3 are whether or not they solved 3 different problems correctly and the underlying concept is "problem solving ability" or "intelligence".

In that case I would use a SEM, such that you use measurement models to estimate the underlying latent variables and the relationship between these latent variables.