# What is the best practices/way to do a linear regression on highly correlated variables

I wish to create a composite variable of a number of highly correlated variables. Each one of these variables contains different information and is useful in it's own right. But they are all highly correlated (sometimes 0.8+). Extreme values of overfitting regularly occur during linear regression with one parameter becoming highly negative and another one becoming highly positive, such that the correlations overall net out the two parameter effects.

What is this recommended approach to doing regression on highly correlated variables while avoiding issues such as this caused by correlation?

Previously I was constraining all coefficients to be positive or zero. However I am moving things over to R and only a limited number of packages seem to support constraints. So I wanted to know if there are better ways of approaching this? Regularisation or selection would probably solve some of the problem, although they wouldn't rule out some spurious values coming through because of the correlations. Another possibility I thought of was using PCA with all components being used in the regression?

If it's of interest, one of the examples would be using financial data to create a composite Value indicator using PE ratios, P/CF ratios and P/Sales ratios, each of which contains different but closely related information. It would add another couple of ratios in, and include relative to industry and absolute versions of each, so I am looking at about 12 or so different variables with correlations often between 0.8 to over 0.9.

• L1 or L2 regularization with cross-validation would be a natural way to go about this – Arthur B. Nov 14 '14 at 16:34
• Is your goal to interpret the effects of these variables, or to use them to predict another variable? – Sean Easter Nov 14 '14 at 20:59
• To predict another variable. In the example of the Value indicator it would be to predict future price movements – Graeme Nov 16 '14 at 11:00

• A lot of people (myself included) seem to get good mileage out of the glmnet package as well – shadowtalker Dec 28 '14 at 16:10