10
$\begingroup$

I'm reading this interesting paper on application of ICA to gene expression data.

The authors write:

[T]here is no requirement for PCA components to be statistically independent.

That is true, but the PCs are orthogonal, are they not?

I am a bit fuzzy as to what is the relationship between statistical inedpendence and orthogonality or linear independence.

It is worth noting that while ICA also provides a linear decomposition of the data matrix, the requirement of statistical independence implies that the data covariance matrix is decorrelated in a non-linear fashion, in contrast to PCA where the decorrelation is performed linearly.

I don't understand that. How does lack of linearity follow from statistical independence?

Question: how does statistical independence of components in ICA relate to linear independence of components in PCA?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
15
$\begingroup$

This is likely to be a duplicate of some older question(s), but I will briefly answer is nevertheless.

For a non-technical explanation, I find quite helpful this figure from the Wikipedia article on Correlation and dependence:

enter image description here

The numbers above each scatter plot show correlation coefficients between X and Y. Look at the last row: on each scatter plot the correlation is zero, i.e. X and Y are "linearly independent". However they are obviously not statistically independent: if you know the value of X, you can narrow down the possible values of Y. If X and Y were independent, it would mean that knowing X tells you nothing about Y.

The purpose of ICA is to try to find independent components. In PCA you only get uncorrelated ("orthogonal") components; correlation between them is zero but they can very well be statistically dependent.

Improve this answer
$\endgroup$
5
  • 2
    $\begingroup$ Ah! (palmface) OK, somehow I started dissecting the ICA and ended up not seeing the obvious. Thanks! I use the same example when explaining the same problem to others... $\endgroup$
    – January
    Commented Dec 18, 2014 at 16:00
  • 1
    $\begingroup$ We tend to "equate" "orthogonality" with "zero correlation", but this is true only when one of the variables involved has zero mean. $\endgroup$
    – Alecos Papadopoulos
    Commented Dec 18, 2014 at 17:46
  • 2
    $\begingroup$ @Alecos, that is correct (+1), but analyses such as PCA or ICA are almost always done on centered variables, so this distinction is not relevant. $\endgroup$
    – amoeba
    Commented Dec 18, 2014 at 19:35
  • 1
    $\begingroup$ Indeed, that's the problem in general. In econometrics, orthogonality is discussed mostly with respect to the "error term" of a regression which has zero-mean, and so here too, it tends to be equated with "zero covariance". So people run the danger to forget that in general they are not equal, and so they may end up wrongly assume so in a situation where the variables are not centered to their mean. $\endgroup$
    – Alecos Papadopoulos
    Commented Dec 18, 2014 at 19:47
  • $\begingroup$ I encountered a sentence: "Though uncorrelated, the principal components can be highly statistically dependent". Following your answer, is it reasonable to understand it in the following way: knowing what one PC is, we are able to tell something about a different PC? $\endgroup$
    – kamilazdybal
    Commented Feb 11, 2019 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.