I recently came across one of the old Kaggle challenges, where the participants found out that there were numerous highly correlated variables (>0.95) and had to delete one of them from data modeling. However, there was a twist in this competition. The difference between the two highly correlated variables (both standardized), turned out to be a very good predictor as well. The participants called it "Golden Feature". You can read more in the discussion forum but aforementioned is basically what they did.

Now, after going through all the discussion in Kaggle, I couldn't find out any underlying statistical theory behind it. On the other hand, it seems to me a valid way to handle multicollinearity issue. Because keeping the difference would mean we are not losing any information by deleting original correlated variable.

I can't find any scientific article on this, which implies I am most probably wrong. Can anyone help me out pointing out my mistake? Also, how to deal with highly correlated variables then?

  • $\begingroup$ What is your question exactly? Is the question "Can I handle multicollinearity through adding/taking the difference of the variables"? $\endgroup$
    – dietervdf
    Aug 7, 2018 at 23:07
  • $\begingroup$ That would be the gist of it. $\endgroup$ Aug 8, 2018 at 9:12

1 Answer 1


One of the main methods of dealing with multicollinearity is through PCA (principal components analysis), which uses a linear combination of predictors. You might want to take a look at that.

Principal Component Analysis to Address Multicollinearity (Lexi V. Perez) seems to contain a simple explanation.

To me it's not really clear why the Kaggle project deals with the issue multicollinearity since they want to construct a prediction model (right?) and are not interested in the (conditional) effect estimate of individual predictors. To my knowledge multicollinearity has no big effect on prediction. (Overfitting might be an issue since the variable selection could be distorted)

Do you know the issues of multicollinearity? Let me illustrate one of the issues using a simple example.

Assume builds a model which estimates cholesterol levels based on different predictors like age, gender, weight, height, etc...

Since weight and height are highly correlated, multicollinearity would distort the effect estimates of weight and height. Say the model results in $$\mathrm{Cholesterol = \ldots + 0.5 \cdot \mathrm{Weight}+0.7 \cdot\mathrm{Height}+ \ldots}$$

Usually you would interpret the effect estimates as: 'When all other covariates are kept constant, an increase of weight by 10 increases the cholesterol levels by 5'. However the multicollinearity issue could result in a biased effect estimate which would mean that the effect estimate 0.5 is useless... (Think like: it is impossible to keep the height constant when the weight increases by 10, so that interpretation is no longer valid).

Often this issue is resolved through the usage of BMI. Combining the effect of weight and height (not in a linear fashion though, but still). Then the model would result in a effect estimate of BMI, which could be interpreted.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.