I read that correlation is used to identify the relationship between variables and I have a few questions about the same, I was hoping someone could help me answer them.

  1. what is the use of knowing the relationship between correlated variables? Is it used in machine learning to identify and drop features that are highly related to reduce the feature space?
  2. Does removing highly correlated variables improve performance of the model and if yes, how?
  3. Is there anything else that it can be used for?

You can view correlation as how much information variables share. Note also that correlation can occur simply by chance.

Imagine that your you collect a sample with three variables: $X_1, X_2$ and $Y$ and that the true underlying function is $Y=c X_1$ for some constant c.

If you have a very high correlation between $X_1$ and $X_2$ in your sample then it will be hard to distinguish the true underlying model as both $X_1$ and $X_2$ "competes" against each other and in your sample thus behaves the same. That is either variable will be equally good to predict $Y$ in your sample due to the correlation, but only $X_1$ is truly good.


All three of your questions are yes. Feature importance is one of the most significant purposes of machine learning. Including features that are highly correlated with one another is almost always avoided because it is better to only include one representative from a correlated group of features in your model while discarding the other ones that are deemed to be redundant. Using only one representative feature per group is known to enhance performance and essentially underlies the motivation of principal component analysis (PCA), which is a dimensionality reduction technique, and ridge regression and lasso.

Other usages of correlation analysis of machine learning besides feature importance can probably found in the functionality of clustering algorithms which are unsupervised learning models that employ distance metrics to group blobs of data together based on their similarities. The correlation that exists amongst the features guides the clustering process.

  1. It might help. All else equal, we tend to prefer simpler models. If you can drop a variable whose influence on the outcome is mostly contained in another variable, perhaps the increase in in-sample performance is not worth the hit of additional model complexity. In the extreme case, consider a variable where the units are in meters and another variable with the same measurements but in feet. You get no new information by including both instead of one, so it would not be worth the increased model complexity to include both.
  • $\begingroup$ if you get no new information $\endgroup$ – Tim Aug 4 '20 at 13:49
  • $\begingroup$ @Tim do you see a way that the feet and meters variables could contain unique information? $\endgroup$ – Dave Aug 4 '20 at 14:08
  • 2
    $\begingroup$ But those are perfectly collinear, not just correlated. The point is that people often use correlation like "oh, the correlation is 0.87, so let's drop one of those". $\endgroup$ – Tim Aug 4 '20 at 14:12
  • 1
    $\begingroup$ To add to @Tim's point...if you had separate, noisy measurements in feet and meters (e.g. from two different rangefinder devices), you would indeed gain additional information over a single measurement, despite the correlation $\endgroup$ – user20160 Aug 4 '20 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.