# ICA - independence of coefficients and maximizing independence

Hopefully this isn't too silly a question but I'm wondering how in independent component analysis when we've got independent coefficients then we identify parts of a face such as eyes, mouth, nose, etc.

What exactly does the independence of the coefficients have to do with this? Isn't it more to do with the independence of the proposed feature vectors?

Also, is the primary difference in PCA and ICA that PCA aims to maximize the variance of it's projections which is given by the directions provided by the eigendecomposition of its covariance matrix and ICA aims to maximize the independence of its feature vectors? But since the the covariance matrix is a symmetrical matrix, aren't all the eigenvectors independent anyway?

How exactly does ICA maximize independence in a way that makes them more independent than the orthogonal basis vectors produced by PCA?

Thanks

how in independent component analysis when we've got independent coefficients then we identify parts of a face such as eyes, mouth, nose, etc.

My interpretation of your question is: Suppose we run ICA on a set of aligned face images, treating pixels as dimensions and images as data points. Why do weight vectors (i.e. basis images) produced by ICA tend to assign high weight to groups of pixels that correspond to facial features--is that what you're asking? The tautological answer would be something like "because that's how the distribution factorizes best, subject to the assumptions of ICA". Loosely, you could say this means that pixels within each facial feature tend to vary together across faces, and to vary independently from those in other facial features. For this to be true, the face images would probably have to be aligned such that the same set of pixels consistently covers the same facial features across images.

Also, is the primary difference in PCA and ICA that PCA aims to maximize the variance of it's projections which is given by the directions provided by the eigendecomposition of its covariance matrix and ICA aims to maximize the independence of its feature vectors?

Your statement about PCA is correct. I'm not sure I understand the precise intended meaning of your statement about ICA, so I'll phrase it another way: ICA tries to maximize the independence of the projections of the data onto each weight/basis vector. Note that this is distinct from independence of the basis vectors themselves. ICA typically assumes that the data are i.i.d., that there's no noise, and that the number of components is equal to the number of input dimensions. PCA doesn't share the last assumption.

But since the the covariance matrix is a symmetrical matrix, aren't all the eigenvectors independent anyway?

If the data has $d$ dimensions and the covariance matrix has full rank, there will be $d$ linearly independent eigenvectors. A set of vectors is linearly independent if it's impossible to write one of the vectors as a linear combination of the others. This is true of the eigenvectors because they're orthogonal. Note that linear independence is a distinct concept from statistical independence, and linear independence of the basis vectors has nothing to do with ICA. Unlike PCA, ICA doesn't require the basis vectors to be orthogonal. Furthermore, it tries to maximize statistical independence of the projections, not the basis vectors.

• Oh man, this was really helpful. I was completely missing the part about statistical independence vs linear independence as it wasn't specifically stated in the notes. Just a follow up: that statistical independence is therefore determined by the correlation coefficients between two points being zero is that right? So in ICA, are we looking for a set of basis vectors that produce approximations where the approximations for point x across basis vector v1 and across basis vector v2 are statistically independent. – tryingtolearn Mar 26 '17 at 17:42
• If there were two dimensions/components, then ICA would try to find basis vectors v1 and v2 s.t. the projections of the data onto v1 are maximally independent from the projections onto v2. But, this isn't equivalent to minimizing the correlation coefficient, because zero correlation doesn't imply independence. Ideally, we'd like to maximize the joint entropy. But, that may not be feasible to estimate, so different forms of ICA optimize different surrogate loss functions instead. – user20160 Mar 26 '17 at 18:31
• I definitely recommend reading some ICA papers. For example: Hvarinenen and Oja (2000). Independent component analysis: algorithms and applications. – user20160 Mar 26 '17 at 18:32
• Okay, but maximally independence implies that cov(v1,v2) = 0 right? So we want a basis of vectors where this is true for all v1,v2,... etc. – tryingtolearn Mar 26 '17 at 18:43