32
$\begingroup$

A common pre-processing step for machine learning algorithms is whitening of data.

It seems like it is always good to do whitening since it de-correlates the data, making it simpler to model.

When is whitening not recommended?

Note: I'm referring to de-correlation of the data.

$\endgroup$
  • 1
    $\begingroup$ can you give reference for whitening ? $\endgroup$ – Atilla Ozgur Feb 17 '12 at 14:43
  • 3
    $\begingroup$ I think this thread is a stub. It should really be expanded. - - The currently accepted answer has so little piece of information. - - I would unaccept it and open a bounty here. $\endgroup$ – Léo Léopold Hertz 준영 Dec 26 '16 at 13:02
  • $\begingroup$ Your question is also biased, by having "always" there. Of course, whitening is not always good. Also, define types of whitening. I think it leads itself to not so constructive answers here. - - Define types of data to be used. - - I think a better question can be How can you improve the application of this whitening on this nice enough data?. - - @AtillaOzgur One source en.wikipedia.org/wiki/Whitening_transformation if the basic transformation of whitening is considered. $\endgroup$ – Léo Léopold Hertz 준영 Dec 26 '16 at 13:10
  • $\begingroup$ @LéoLéopoldHertz준영 why don't you ask a separate well defined question? (and post a link here so that those who visit this question can also take a look there) $\endgroup$ – Nagabhushan S N Mar 30 at 12:51
14
$\begingroup$

Pre-whitening is a generalization of feature normalization, which makes the input independent by transforming it against a transformed input covariance matrix. I can't see why this may be a bad thing.

However, a quick search revealed "The Feasibility of Data Whitening to Improve Performance of Weather Radar" (pdf) which reads:

In particular, whitening worked well in the case of the exponential ACF (which is in agreement with Monakov’s results) but less well in the case of the Gaussian one. After numerical experimentation, we found that the Gaussian case is numerically ill conditioned in the sense that the condition number (ratio of maximal to minimal eigenvalue) is extremely large for the Gaussian covariance matrix.

I'm not educated enough to comment on this. Maybe the answer to your question is that whitening is always good but there are certain gotchas (e.g., with random data it won't work well if done via Gaussian autocorrelation function).

| cite | improve this answer | |
$\endgroup$
  • 3
    $\begingroup$ as I understand it, it works well if the covariance matrix is well-estimated. Can someone comment on this? thanks. $\endgroup$ – Ran Feb 15 '12 at 15:10
  • 3
    $\begingroup$ The quote above is not referring to a poorly estimated covariance matrix (although that would also be problematic). It is saying that for a perfectly specified covariance matrix, it can still be difficult to accurately perform the required factorization (and associated data transformations). This is due to numerical ill-conditioning, which means finite-precision roundoff errors pollute the computations. $\endgroup$ – GeoMatt22 Sep 9 '16 at 2:12
  • 3
    $\begingroup$ This is insufficient answer. It has mostly copied not-so-related material. - - This answer should really be expanded. It is a stub. $\endgroup$ – Léo Léopold Hertz 준영 Dec 26 '16 at 13:03
27
$\begingroup$

Firstly, I think that de-correlating and whitening are two separate procedures.

In order to de-correlate the data, we need to transform it so that the transformed data will have a diagonal covariance matrix. This transform can be found by solving the eigenvalue problem. We find the eigenvectors and associated eigenvalues of the covariance matrix ${\bf \Sigma} = {\bf X}{\bf X}'$ by solving

$${\bf \Sigma}{\bf \Phi} = {\bf \Phi} {\bf \Lambda}$$

where ${\bf \Lambda}$ is a diagonal matrix having the eigenvalues as its diagonal elements.

The matrix ${\bf \Phi}$ thus diagonalizes the covariance matrix of ${\bf X}$. The columns of ${\bf \Phi}$ are the eigenvectors of the covariance matrix.

We can also write the diagonalized covariance as:

$${\bf \Phi}' {\bf \Sigma} {\bf \Phi} = {\bf \Lambda} \tag{1}$$

So to de-correlate a single vector ${\bf x}_i$, we do:

$${\bf x}_i^* = {\bf \Phi}' {\bf x}_i \tag{2}$$

The diagonal elements (eigenvalues) in ${\bf \Lambda}$ may be the same or different. If we make them all the same, then this is called whitening the data. Since each eigenvalue determines the length of its associated eigenvector, the covariance will correspond to an ellipse when the data is not whitened, and to a sphere (having all dimensions the same length, or uniform) when the data is whitened. Whitening is performed as follows:

$${\bf \Lambda}^{-1/2} {\bf \Lambda} {\bf \Lambda}^{-1/2} = {\bf I}$$

Equivalently, substituting in $(1)$, we write:

$${\bf \Lambda}^{-1/2} {\bf \Phi}' {\bf \Sigma} {\bf \Phi} {\bf \Lambda}^{-1/2} = {\bf I}$$

Thus, to apply this whitening transform to ${\bf x}_i^*$ we simply multiply it by this scale factor, obtaining the whitened data point ${\bf x}_i^\dagger$:

$${\bf x}_i^{\dagger} = {\bf \Lambda}^{-1/2} {\bf x}_i^* = {\bf \Lambda}^{-1/2}{\bf \Phi}'{\bf x}_i \tag 3$$

Now the covariance of ${\bf x}_i^\dagger$ is not only diagonal, but also uniform (white), since the covariance of ${\bf x}_i^\dagger$, ${\bf E}({\bf x}_i^\dagger {{\bf x}_i^\dagger}') = {\bf I}$.

Following on from this, I can see two cases where this might not be useful. The first is rather trivial, it could happen that the scaling of data examples is somehow important in the inference problem you are looking at. Of course you could the eigenvalues as an additional set of features to get around this. The second is a computational issue: firstly you have to compute the covariance matrix ${\bf \Sigma}$, which may be too large to fit in memory (if you have thousands of features) or take too long to compute; secondly the eigenvalue decomposition is O(n^3) in practice, which again is pretty horrible with a large number of features.

And finally, there is a common "gotcha" that people should be careful of. One must be careful that you calculate the scaling factors on the training data, and then you use equations (2) and (3) to apply the same scaling factors to the test data, otherwise you are at risk of overfitting (you would be using information from the test set in the training process).

Source: http://courses.media.mit.edu/2010fall/mas622j/whiten.pdf

| cite | improve this answer | |
$\endgroup$
  • 3
    $\begingroup$ Thanks for the clarification, you are right. I was referring to de-correlating. btw: at the end you write that whitening is only performed to the training data. as far as I know, you compute the matrix from the training data, but you perform it on both training & test data. $\endgroup$ – Ran Feb 17 '12 at 13:04
  • $\begingroup$ @Ran yes that's what I meant ... I'll update the answer $\endgroup$ – tdc Feb 17 '12 at 14:36
  • $\begingroup$ It would be nice if you could also offer sections in your answer. Have a intro, a summary and the math things. - - I think you do not go deep enough in your answer. - - Your answer covers mostly trivial propositions but does not go deep enough in the topic. You have just basic copy-pasted material from lecture notes but very little own work for the topic. $\endgroup$ – Léo Léopold Hertz 준영 Dec 26 '16 at 13:04
  • $\begingroup$ so in simple terms, do pca to get de-correlated features, and then foreach new feature, divide by the variance to get whitened features. $\endgroup$ – avocado Nov 9 '17 at 7:19
  • 2
    $\begingroup$ This should be the accepted answer - and the first 'trivial' case on non-usefulness should be emphasized: whitening can harm performance, big time. One example is multivariate signal classification; relative amplitudes and (dc) biases may carry important information that'd be destroyed by whitening. $\endgroup$ – OverLordGoldDragon Dec 12 '19 at 22:35
3
$\begingroup$

From http://cs231n.github.io/neural-networks-2/

One weakness of this transformation is that it can greatly exaggerate the noise in the data, since it stretches all dimensions (including the irrelevant dimensions of tiny variance that are mostly noise) to be of equal size in the input. This can in practice be mitigated by stronger smoothing...

Unfortunately I'm not educated enough to comment further on this.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Please, state which forms of noises are exaggerated. Your reference is rigorous. It is just basic computer science about the topic i.e. white noise with an ancient neural network approach. - - The work exaggerate should also be defined. $\endgroup$ – Léo Léopold Hertz 준영 Dec 26 '16 at 13:07
  • $\begingroup$ Seems to me that this is just related to the scaling of all features to have the same variance, right? So if there were a feature whose variance in the training set were noise, we might expect the overall variance of this feature to be much smaller than another feature; this transformation would make both the "noise" feature and the other feature have the same variance, and could be seen as "amplifying noise". $\endgroup$ – ijoseph Jun 19 '17 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.