5
$\begingroup$

I read that correlation is used to identify the relationship between variables and I have a few questions about the same, I was hoping someone could help me answer them.

  1. what is the use of knowing the relationship between correlated variables? Is it used in machine learning to identify and drop features that are highly related to reduce the feature space?
  2. Does removing highly correlated variables improve performance of the model and if yes, how?
  3. Is there anything else that it can be used for?
$\endgroup$

2 Answers 2

2
$\begingroup$

You can view correlation as how much information variables share. Note also that correlation can occur simply by chance.

Imagine that your you collect a sample with three variables: $X_1, X_2$ and $Y$ and that the true underlying function is $Y=c X_1$ for some constant c.

If you have a very high correlation between $X_1$ and $X_2$ in your sample then it will be hard to distinguish the true underlying model as both $X_1$ and $X_2$ "competes" against each other and in your sample thus behaves the same. That is either variable will be equally good to predict $Y$ in your sample due to the correlation, but only $X_1$ is truly good.

$\endgroup$
1
$\begingroup$

All three of your questions are yes. Feature importance is one of the most significant purposes of machine learning. Including features that are highly correlated with one another is almost always avoided because it is better to only include one representative from a correlated group of features in your model while discarding the other ones that are deemed to be redundant. Using only one representative feature per group is known to enhance performance and essentially underlies the motivation of principal component analysis (PCA), which is a dimensionality reduction technique, and ridge regression and lasso.

Other usages of correlation analysis of machine learning besides feature importance can probably found in the functionality of clustering algorithms which are unsupervised learning models that employ distance metrics to group blobs of data together based on their similarities. The correlation that exists amongst the features guides the clustering process.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.