One of the steps in performing whitening is decorrelation. I understand that decorrelation reduces the correlation among various input features, but haven't found a compelling reason why reduced correlation helps machine learning models perform better most of the times.
Is there any mathematical or intuitive explanation to explain this or it's mostly an empirical finding?
Let's understand why correlated variables are removed when fitting ML models.
If 2 or more variables are perfectly correlated with each other, then it makes sense to remove them and let only one of them stay, as all of them span/explain the same dimension, thus, not adding any additional information.
So, when your data is considerably large, you might want to decorrelate the dataset before fitting an ML model, so that the size can be trimmed which would result in the model training and performing faster due to less number of columns.
Also, some parametric models carry the risk of being erratic due to their covariance matrix becoming singular (Reason: Due to multiple columns spanning the same dimension).
Inputing 2 (and more) correlated features (known to be multicollinear features) to the model you inflate variance of the model's output but in order to create good model you should strive to dicrease variance as much as possible to get best linear unbiased estimator - BLUE
Remember, that BLUE estimates should be:
Whitening is a useful preprocessing step because it both decorrelates and normalises the inputs -- is used i.e. in sklearn's PCA (dimensionality reduction) method by setting param whiten=True
