I wish to create a composite variable of a number of highly correlated variables. Each one of these variables contains different information and is useful in it's own right. But they are all highly correlated (sometimes 0.8+). Extreme values of overfitting regularly occur during linear regression with one parameter becoming highly negative and another one becoming highly positive, such that the correlations overall net out the two parameter effects.

What is this recommended approach to doing regression on highly correlated variables while avoiding issues such as this caused by correlation?

Previously I was constraining all coefficients to be positive or zero. However I am moving things over to R and only a limited number of packages seem to support constraints. So I wanted to know if there are better ways of approaching this? Regularisation or selection would probably solve some of the problem, although they wouldn't rule out some spurious values coming through because of the correlations. Another possibility I thought of was using PCA with all components being used in the regression?

If it's of interest, one of the examples would be using financial data to create a composite Value indicator using PE ratios, P/CF ratios and P/Sales ratios, each of which contains different but closely related information. It would add another couple of ratios in, and include relative to industry and absolute versions of each, so I am looking at about 12 or so different variables with correlations often between 0.8 to over 0.9.

  • 3
    $\begingroup$ L1 or L2 regularization with cross-validation would be a natural way to go about this $\endgroup$
    – Arthur B.
    Nov 14 '14 at 16:34
  • $\begingroup$ Is your goal to interpret the effects of these variables, or to use them to predict another variable? $\endgroup$ Nov 14 '14 at 20:59
  • $\begingroup$ To predict another variable. In the example of the Value indicator it would be to predict future price movements $\endgroup$
    – Graeme
    Nov 16 '14 at 11:00

I would give the correct answer to Arthur B. This is more Stack Overflow than Cross Validated, but the solution I found was to use the penalized package in R which combines L1 and L2 regularization. This has a parameter that gives the option of only having positive coefficients which was what I was looking for. But in playing around with it I got better results by ignoring this parameter and just relying on the regularization.

  • $\begingroup$ A lot of people (myself included) seem to get good mileage out of the glmnet package as well $\endgroup$ Dec 28 '14 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.