Whitening/Decorrelation - why does it work?

Question

Given some whitening transform, we change some vectors $\textbf{x}$, where features are correlated, into some vector $\textbf{y}$, where components are uncorrelated. Then we run some learning algorithm on the transformed vectors $y$.

Why does this work? In the original space we had correlation between vector components, and it carried some information (correlation IS information, right?). Now, we whiten data and get a round blob as output. All information about correlation lost - to me seems like a large part of information lost. Wouldn't it be much easier to learn decision boundaries/distributions defined over correlated data (since ALL information present)? So, how does SVM (since in my practice this is the method that requires this most), for example, gives a better result with whitening?

Answer 1

Whitening is par for the course in computer vision applications, and may help a variety of machine learning algorithms converge to an optimal solution, beyond SVMs. (More on that towards the end of my answer.)

Now, we whiten data and get a round blob as output.

To put that more mathematically, whitening transforms a distribution using its eigenvectors $\boldsymbol{u}_j$ in such a way that its covariance matrix becomes the unit matrix. Bishop (1995) pp. 300 illustrates this in a stylised manner:

Whitening is a useful preprocessing step because it both decorrelates and normalises the inputs.

Decorrelation

The training step of machine learning algorithms is simply an optimisation problem, however it is defined. Whitening gives nice optimisation properties to the input variables, causing such optimisation steps to converge faster. The mechanism for this improvement is that it affects the condition number of the Hessian in steepest descent-style optimisation algorithms. Here are some sources for further reading:

• Maurya's answer on Quora
• Smith - Conditioning and Hessians in analytical and numerical optimisation: Some illustrations

Normalisation

The fact that input variables now have unit variance is an example of feature normalisation, which is a prerequisite for many ML algorithms. Indeed, SVMs (along with regularized linear regression and neural networks) requires that features be normalised to work effectively, so whitening may be improving your SVMs' performance significantly thanks only to the feature normalisation effect.

Answer 2

This is a couple years late, but I think this is a very good question and there are not many clear intuitive answers about it. I could be wrong about some of this, so if someone could verify this, that would be great.

What Does Whitening Do?

There are a good amount of images online that show this, but the way I see it, there are two main components of whitening: decorrelation and normalization.

Normalization is just enforcing unit variance. No matter the application, this is almost always good because different scales between different variables can make your answers uninterpretable or hard to solve.

Decorrelation is the main aspect which causes us to lose information. However, the information we are losing, is information we do not want (given you are using whitening when you should). Decorrelation removes the shared components of our sources/data. Specifically, it removes the first and second moments (mean and variance/covariance granted your data is demeaned as well). In many applications of machine learning this is beneficial. This is because with methods like ICA or neural nets we are no longer concerned with low order statistics/linear relationships. That is the reason we are doing these methods in the first place. If our model was clearly linear, we could use a linear regression (in this scenario you definitely would not want to whiten because you would be erasing the linear relationships which would be very counter intuitive).

For neural nets, this decorrelation is beneficial because we are making our data more linearly independent and separable. This allows our model to better train on the differences between our data. If after we whiten, there is no more higher order information, that implies that this dataset might not be good for your model or vice versa (maybe just use linear regression instead).

For ICA, whitening is beneficial because we assume our final components will be independent. By whitening our data, we are making the covariance matrix the identity which will be true for our final answer because the components are independent. The whitened data using PCA for ICA can almost be viewed as an informed prior or initial rough estimate of our final answer. It takes our data (separate signals) and models them as sources that are uncorrelated. Then the rest of the ICA algorithm needs to make them independent (note uncorrelated/linear independence does not imply statistical independence).

In summary whitening can have different but similar benefits depending on the application. However, before you whiten, you should make sure you do not care about the linear correlations in your data.

Understanding the Scatter Plots The scatter plots you are talking about usual plot the data where one component is the x axis, and the other component is the y axis. The points are correlated points (maybe in time or space).

Ex. Data Collection: You have two microphones, Mic A and Mic B, recording audio signals simultaneously. Each microphone captures sound from the environment. Over time, they generate a sequence of audio samples. Creation of Scatter Plot: To create the scatter plot, you pair up the audio samples from Mic A and Mic B at each time point. Each pair of samples represents a single point on the scatter plot. The X-axis of the scatter plot corresponds to the amplitude or value of the audio signal from Mic A at that moment, while the Y-axis corresponds to the value of the audio signal from Mic B at the same moment. Scatter Plot Points: The points on the scatter plot represent these paired values. If there is some degree of correlation or dependency between the signals recorded by the two microphones, you may observe a pattern in the scatter plot. For example, if the microphones are close together and both capture the same sound source, you might see a linear correlation in the scatter plot because the two signals are similar.

So, after whitening you see that our data is a "blob". This is because we are relating our two components in linear manner. Since we just erased linear correlations, we would expect that it should just look like a blob or white noise i.e., the graph is saying there is no more linear information contained. However, if you visualize it differently or correlate your components in a nonlinear manner you might see different correlations or relationships.

One thing I was confused about initially with whitening for ICA was, "why do we want to decorrelate our signals? the correlation between different signal is what is helping us find the sources." But then I realized what was being plotted or the resulting data after PCA was not really a representation of our recorded signals, but better thought of as a rough estimation of our decorrelated sources. If we were to plot our sources that we are trying to estimate in a similar manner, we would expect to see a blob as well.