2
$\begingroup$

One of the steps in performing whitening is decorrelation. I understand that decorrelation reduces the correlation among various input features, but haven't found a compelling reason why reduced correlation helps machine learning models perform better most of the times.

Is there any mathematical or intuitive explanation to explain this or it's mostly an empirical finding?

$\endgroup$
2
3
$\begingroup$

Let's understand why correlated variables are removed when fitting ML models.

If 2 or more variables are perfectly correlated with each other, then it makes sense to remove them and let only one of them stay, as all of them span/explain the same dimension, thus, not adding any additional information.

So, when your data is considerably large, you might want to decorrelate the dataset before fitting an ML model, so that the size can be trimmed which would result in the model training and performing faster due to less number of columns.

Also, some parametric models carry the risk of being erratic due to their covariance matrix becoming singular (Reason: Due to multiple columns spanning the same dimension).

$\endgroup$
1
  • $\begingroup$ Given the fact that correlation only captures second order statistics, isn't it better to find independent features instead of uncorrelated features then? $\endgroup$ – Matt Aug 11 '17 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.