2
$\begingroup$

I know there are a couple posts asking about why we whiten the data for ICA. I understand why we whiten to fix scaling invariants between the sources and to increase the computationally efficiency.

But most answers mention something saying we want to decorrelate our data before running ICA or it should be orthogonal. Could someone please explain the intuition behind this as well as the example in this link https://arnauddelorme.com/ica_for_dummies/.

Specifically, why does our data need to be decorrelated? The ICA assumption is that our data is statistically independent. Decorrelating our data does not create statistical independence just a form of linear independence.

As for the example in the link, I do not understand it all. If someone could break it down clearly that would be a huge help. One of the many questions I have about it is why do they apply a linear transformation to the data in the example.

A clear geometric intuition would be beyond appreciated. Thank you for the clarification.

Edit/Update

I saw someone else had a similar question, so after I did some research and felt like I had a decent intuition, I provided an explanation that I think should clear up both questions. It can be found here Whitening/Decorrelation - why does it work?.

Please take all of this with a grain of salt until someone can confirm it.

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
2
$\begingroup$

The TL;DR is that whitening isn't essential but it does simplify the task. Specifically, whitening the data reduces the number of parameters under estimation.

"Independent Component Analysis: Algorithms and Applications" by Aapo Hyvärinen and Erkki Oja. Neural Networks, 13(4-5):411-430, 2000.

Here we see that whitening reduces the number of parameters to be estimated. Instead of having to estimate the $n^ 2$ parameters that are the elements of the original matrix $A$, we only need to estimate the new, orthogonal mixing matrix $\tilde{A}$. An orthogonal matrix contains $n(n − 1)/2$ degrees of freedom. For example, in two dimensions, an orthogonal transformation is determined by a single angle parameter. In larger dimensions, an orthogonal matrix contains only about half of the number of parameters of an arbitrary matrix. Thus one can say that whitening solves half of the problem of ICA. Because whitening is a very simple and standard procedure, much simpler than any ICA algorithms, it is a good idea to reduce the complexity of the problem this way.

The geometric intuition is that whitening doesn't remove any geometric information. It just rotates the data. Correlations only make sense in a specific coordinate system; if we rotate the coordinates, we can remove correlation.

enter image description here The second row of this diagram taken from Wikipedia contains several lines, but the center one has 0 correlation, even though it's just a rotation of the other lines in that row.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thank you for the reply. A couple of follow ups 1. "The geometric intuition is that whitening doesn't remove any information." This is not necessarily, true right? Specifically, we are removing the first two moments (mean and covariance). However, in the case of ICA, this is fine because our final answer is statistically independent so it should ultimately have 0 covariance. I just think it might be important to make this distinction that whitening does cause a loss of information because in other contexts (other ML models) it might be wise to think twice about whitening. $\endgroup$
    – user19402204
    Sep 26 at 20:08
  • $\begingroup$ but in the case of ICA we can view whitening as killing two birds with one stone: remove scaling invariance from each component and also give us an uncorrelated representation of our data which can be thought of as an informed prior or rough estimate of our final answer. 2. Secondly, I am still not understanding the visual in the link I put in. I get the scatter plots Ive seen on other sources that show after whitening we just have a circle or blob. $\endgroup$
    – user19402204
    Sep 26 at 20:11
  • $\begingroup$ This makes sense, we have removed linear relationships so we now should not be able to see any linear correlation between the variables in the scatter plots. However, I have no clue what is going on in that link so if someone could break that down, that would be incredibly helpful. Thanks $\endgroup$
    – user19402204
    Sep 26 at 20:14
  • $\begingroup$ The mean and variance aren't geometric information. Take the axis labels off of a plot of the data and it will look the same if you shift it or rescale it by a positive scalar. // The only question in your post is about the geometric intuition of whitening for ICA. If you're specifically interested in understanding a particular portion of the link, you should post a new question quoting the passage you want to know about and clearly articulate what part you're having trouble understanding. $\endgroup$
    – Sycorax ♦
    Sep 26 at 20:24
  • $\begingroup$ That makes sense, thanks! Just to confirm, you do lose information but not necessarily geometric information (and the statistical information lost in ICA does not make a difference)? $\endgroup$
    – user19402204
    Sep 26 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.