2
$\begingroup$

I wish to create a composite variable of a number of highly correlated variables. Each one of these variables contains different information and is useful in it's own right. But they are all highly correlated (sometimes 0.8+). Extreme values of overfitting regularly occur during linear regression with one parameter becoming highly negative and another one becoming highly positive, such that the correlations overall net out the two parameter effects.

What is this recommended approach to doing regression on highly correlated variables while avoiding issues such as this caused by correlation?

Previously I was constraining all coefficients to be positive or zero. However I am moving things over to R and only a limited number of packages seem to support constraints. So I wanted to know if there are better ways of approaching this? Regularisation or selection would probably solve some of the problem, although they wouldn't rule out some spurious values coming through because of the correlations. Another possibility I thought of was using PCA with all components being used in the regression?

If it's of interest, one of the examples would be using financial data to create a composite Value indicator using PE ratios, P/CF ratios and P/Sales ratios, each of which contains different but closely related information. It would add another couple of ratios in, and include relative to industry and absolute versions of each, so I am looking at about 12 or so different variables with correlations often between 0.8 to over 0.9.

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ L1 or L2 regularization with cross-validation would be a natural way to go about this $\endgroup$
    – Arthur B.
    Commented Nov 14, 2014 at 16:34
  • $\begingroup$ Is your goal to interpret the effects of these variables, or to use them to predict another variable? $\endgroup$
    – Sean Easter
    Commented Nov 14, 2014 at 20:59
  • $\begingroup$ To predict another variable. In the example of the Value indicator it would be to predict future price movements $\endgroup$
    – Graeme
    Commented Nov 16, 2014 at 11:00

1 Answer 1

Reset to default
0
$\begingroup$

I would give the correct answer to Arthur B. This is more Stack Overflow than Cross Validated, but the solution I found was to use the penalized package in R which combines L1 and L2 regularization. This has a parameter that gives the option of only having positive coefficients which was what I was looking for. But in playing around with it I got better results by ignoring this parameter and just relying on the regularization.

Improve this answer
$\endgroup$
1
  • $\begingroup$ A lot of people (myself included) seem to get good mileage out of the glmnet package as well $\endgroup$
    – shadowtalker
    Commented Dec 28, 2014 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.