2
$\begingroup$

One of the steps in performing whitening is decorrelation. I understand that decorrelation reduces the correlation among various input features, but haven't found a compelling reason why reduced correlation helps machine learning models perform better most of the times.

Is there any mathematical or intuitive explanation to explain this or it's mostly an empirical finding?

$\endgroup$
3

2 Answers 2

3
$\begingroup$

Let's understand why correlated variables are removed when fitting ML models.

If 2 or more variables are perfectly correlated with each other, then it makes sense to remove them and let only one of them stay, as all of them span/explain the same dimension, thus, not adding any additional information.

So, when your data is considerably large, you might want to decorrelate the dataset before fitting an ML model, so that the size can be trimmed which would result in the model training and performing faster due to less number of columns.

Also, some parametric models carry the risk of being erratic due to their covariance matrix becoming singular (Reason: Due to multiple columns spanning the same dimension).

$\endgroup$
1
  • $\begingroup$ Given the fact that correlation only captures second order statistics, isn't it better to find independent features instead of uncorrelated features then? $\endgroup$
    – Matt
    Aug 11, 2017 at 17:27
0
$\begingroup$

Inputing 2 (and more) correlated features (known to be multicollinear features) to the model you inflate variance of the model's output but in order to create good model you should strive to dicrease variance as much as possible to get best linear unbiased estimator - BLUE

Remember, that BLUE estimates should be:

  • unbiased
  • efficient - with lowest variance, which indicates that estimator is the most accurate
  • consistent - Consistency means that as the sample size gets bigger, the sampling error gets smaller.

Whitening is a useful preprocessing step because it both decorrelates and normalises the inputs -- is used i.e. in sklearn's PCA (dimensionality reduction) method by setting param whiten=True

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.