70
$\begingroup$

I am new to Independent Component Analysis (ICA) and have just a rudimentary understanding of the the method. It seems to me that ICA is similar to Factor Analysis (FA) with one exception: ICA assumes that the observed random variables are a linear combination of independent components/factors that are non-gaussian whereas the classical FA model assumes that the observed random variables are a linear combination of correlated, gaussian components/factors.

Is the above accurate?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ This answer to another question (PCA iteratively finds directions of greatest variance; but how to find a whole subspace with greatest variance?) is worth looking at. $\endgroup$
    – Piotr Migdal
    Aug 1, 2015 at 13:04

2 Answers 2

Reset to default
73
$\begingroup$

enter image description here

FA, PCA, and ICA, are all 'related', in as much as all three of them seek basis vectors that the data is projected against, such that you maximize insert-criteria-here. Think of the basis vectors as just encapsulating linear combinations.

For example, lets say your data matrix $\mathbf Z$ was a $2$ x $N$ matrix, that is, you have two random variables, and $N$ observations of them each. Then lets say you found a basis vector of $\mathbf w = \begin{bmatrix}0.1 \\-4 \end{bmatrix}$. When you extract (the first) signal, (call it the vector $\mathbf y$), it is done as so:

$$ \mathbf {y = w^{\mathrm T}Z} $$

This just means "Multiply 0.1 by the first row of your data, and subtract 4 times the second row of your data". Then this gives $\mathbf y$, which is of course a $1$ x $N$ vector that has the property that you maximized its insert-criteria-here.

So what are those criteria?

Second-Order Criteria:

In PCA, you are finding basis vectors that 'best explain' the variance of your data. The first (ie highest ranked) basis vector is going to be one that best fits all the variance from your data. The second one also has this criterion, but must be orthogonal to the first, and so on and so forth. (Turns out those basis vectors for PCA are nothing but the eigenvectors of your data's covariance matrix).

In FA, there is difference between it and PCA, because FA is generative, whereas PCA is not. I have seen FA as being described as 'PCA with noise', where the 'noise' are called 'specific factors'. All the same, the overall conclusion is that PCA and FA are based on second-order statistics, (covariance), and nothing above.

Higher Order Criteria:

In ICA, you are again finding basis vectors, but this time, you want basis vectors that give a result, such that this resulting vector is one of the independent components of the original data. You can do this by maximization of the absolute value of normalized kurtosis - a 4th order statistic. That is, you project your data on some basis vector, and measure the kurtosis of the result. You change your basis vector a little, (usually through gradient ascent), and then measure the kurtosis again, etc etc. Eventually you will happen unto a basis vector that gives you a result that has the highest possible kurtosis, and this is your independent component.

The top diagram above can help you visualize it. You can clearly see how the ICA vectors correspond to the axes of the data, (independent of each other), whereas the PCA vectors try to find directions where variance is maximized. (Somewhat like resultant).

If in the top diagram the PCA vectors look like they almost correspond to the ICA vectors, that is just coincidental. Here is another instance on different data and mixing matrix where they are very different. ;-)

enter image description here

Improve this answer
$\endgroup$
8
  • 2
    $\begingroup$ Seeems that you're familiar with both of the methods. As a competent person, can you answer if those methods inherently imply that the basis vectors are orthogonal? How could one discover the primary or independent components that have a non-zero projection on each other, something like two point clouds oriented approximately at 45 degree angle to each other? $\endgroup$
    – mbaitoff
    Sep 12, 2013 at 7:52
  • 2
    $\begingroup$ @mbaitoff ICA will recover an orthogonal basis set of vectors, yes. Secondly, when you have as you are asking, two signals that have a non-zero projection on each other - that is exactly what ICA is trying to undo. That is why the final basis vectors found by ICA are orthogonal to each other. Then when you project your data on those two new vectors, they are going to be orthogonal to each other. $\endgroup$
    – Spacey
    Sep 12, 2013 at 20:10
  • 1
    $\begingroup$ @Tarantula I have asked a question about what I'm talking: stats.stackexchange.com/questions/6575/… , you can see the illustration, i.stack.imgur.com/U6fWb.png . I cannot understand how an orthogonal basis would describe those two clouds. It's obvious for me that two vectors describing the major oscillation directions are not orthogonal. $\endgroup$
    – mbaitoff
    Sep 13, 2013 at 3:27
  • $\begingroup$ @mbaitoff You took your data from two sensors, and you plot them against each other, and you see those two modes, so you know they are at least correlated. Then the question becomes, how can you project all the points you have there, such that they are independent? (ie, on an orthogonal basis like what ICA finds). That is what ICA finds for you. I dont understand what you mean when you say "I cannot understand how an orthogonal basis would describe those two clouds.". Why not? $\endgroup$
    – Spacey
    Sep 13, 2013 at 5:11
  • $\begingroup$ @Tarantula Oh, now I see what that means! I thought it was like 'finding two orthogonal vectors on original plot' while indeed it means 'finding two vectors on original plot a projection on which will make them orthogonal (independent)'. $\endgroup$
    – mbaitoff
    Sep 13, 2013 at 5:19
32
$\begingroup$

Not quite. Factor analysis operates with the second moments, and really hopes that the data are Gaussian so that the likelihood ratios and stuff like that is not affected by non-normality. ICA, on the other hand, is motivated by the idea that when you add things up, you get something normal, due to CLT, and really hopes that the data are non-normal, so that the non-normal components can be extracted from them. To exploit non-normality, ICA tries to maximize the fourth moment of a linear combination of the inputs:

$$\max_{{\bf a}: \| {\bf a}\| =1} \frac1n \sum_i \bigl[ {\bf a}'({\bf x}_i-\bar {\bf x})\bigr]^4 $$

If anything, ICA should be compared to PCA, which maximizes the second moment (variance) of a standardized combination of inputs.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ nice and crispy answer $\endgroup$
    – user10619
    Mar 14, 2016 at 10:32
  • $\begingroup$ what is the 4th moment here? PL.EXPLAIN. $\endgroup$
    – user10619
    Mar 14, 2016 at 10:34
  • 1
    $\begingroup$ @subhashc.davar 4th moment is kurtosis - i.e the degree to which the data were either heavier or lighter tailed than the normal distribution. en.wikipedia.org/wiki/Kurtosis $\endgroup$
    – WestCoastProjects
    Aug 4, 2016 at 5:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.