Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
A common pre-processing step for machine learning algorithms is whitening of data.
It seems like it is always good to do whitening since it de-correlates the data, making it simpler to model.
When is whitening not recommended?
Note: I'm referring to de-correlation of the data.
Pre-whitening is a generalization of feature normalization, which makes the input independent by transforming it against a transformed input covariance matrix. I can't see why this may be a bad thing.
However, a quick search revealed "The Feasibility of Data Whitening to Improve Performance of Weather Radar" (pdf) which reads:
In particular, whitening worked well in the case of the exponential ACF (which is in agreement with Monakov’s results) but less well in the case of the Gaussian one. After numerical experimentation, we found that the Gaussian case is numerically ill conditioned in the sense that the condition number (ratio of maximal to minimal eigenvalue) is extremely large for the Gaussian covariance matrix.
I'm not educated enough to comment on this. Maybe the answer to your question is that whitening is always good but there are certain gotchas (e.g., with random data it won't work well if done via Gaussian autocorrelation function).
Firstly, I think that de-correlating and whitening are two separate procedures.
In order to de-correlate the data, we need to transform it so that the transformed data will have a diagonal covariance matrix. This transform can be found by solving the eigenvalue problem. We find the eigenvectors and associated eigenvalues of the covariance matrix ${\bf \Sigma} = {\bf X}{\bf X}'$ by solving
$${\bf \Sigma}{\bf \Phi} = {\bf \Phi} {\bf \Lambda}$$
where ${\bf \Lambda}$ is a diagonal matrix having the eigenvalues as its diagonal elements.
The matrix ${\bf \Phi}$ thus diagonalizes the covariance matrix of ${\bf X}$. The columns of ${\bf \Phi}$ are the eigenvectors of the covariance matrix.
We can also write the diagonalized covariance as:
$${\bf \Phi}' {\bf \Sigma} {\bf \Phi} = {\bf \Lambda} \tag{1}$$
So to de-correlate a single vector ${\bf x}_i$, we do:
$${\bf x}_i^* = {\bf \Phi}' {\bf x}_i \tag{2}$$
The diagonal elements (eigenvalues) in ${\bf \Lambda}$ may be the same or different. If we make them all the same, then this is called whitening the data. Since each eigenvalue determines the length of its associated eigenvector, the covariance will correspond to an ellipse when the data is not whitened, and to a sphere (having all dimensions the same length, or uniform) when the data is whitened. Whitening is performed as follows:
$${\bf \Lambda}^{-1/2} {\bf \Lambda} {\bf \Lambda}^{-1/2} = {\bf I}$$
Equivalently, substituting in $(1)$, we write:
$${\bf \Lambda}^{-1/2} {\bf \Phi}' {\bf \Sigma} {\bf \Phi} {\bf \Lambda}^{-1/2} = {\bf I}$$
Thus, to apply this whitening transform to ${\bf x}_i^*$ we simply multiply it by this scale factor, obtaining the whitened data point ${\bf x}_i^\dagger$:
$${\bf x}_i^{\dagger} = {\bf \Lambda}^{-1/2} {\bf x}_i^* = {\bf \Lambda}^{-1/2}{\bf \Phi}'{\bf x}_i \tag 3$$
Now the covariance of ${\bf x}_i^\dagger$ is not only diagonal, but also uniform (white), since the covariance of ${\bf x}_i^\dagger$, ${\bf E}({\bf x}_i^\dagger {{\bf x}_i^\dagger}') = {\bf I}$.
Following on from this, I can see two cases where this might not be useful. The first is rather trivial, it could happen that the scaling of data examples is somehow important in the inference problem you are looking at. Of course you could the eigenvalues as an additional set of features to get around this. The second is a computational issue: firstly you have to compute the covariance matrix ${\bf \Sigma}$, which may be too large to fit in memory (if you have thousands of features) or take too long to compute; secondly the eigenvalue decomposition is O(n^3) in practice, which again is pretty horrible with a large number of features.
And finally, there is a common "gotcha" that people should be careful of. One must be careful that you calculate the scaling factors on the training data, and then you use equations (2) and (3) to apply the same scaling factors to the test data, otherwise you are at risk of overfitting (you would be using information from the test set in the training process).
Source: http://courses.media.mit.edu/2010fall/mas622j/whiten.pdf
From http://cs231n.github.io/neural-networks-2/
One weakness of this transformation is that it can greatly exaggerate the noise in the data, since it stretches all dimensions (including the irrelevant dimensions of tiny variance that are mostly noise) to be of equal size in the input. This can in practice be mitigated by stronger smoothing...
Unfortunately I'm not educated enough to comment further on this.
