how in independent component analysis when we've got independent coefficients then we identify parts of a face such as eyes, mouth, nose, etc.
My interpretation of your question is: Suppose we run ICA on a set of aligned face images, treating pixels as dimensions and images as data points. Why do weight vectors (i.e. basis images) produced by ICA tend to assign high weight to groups of pixels that correspond to facial features--is that what you're asking? The tautological answer would be something like "because that's how the distribution factorizes best, subject to the assumptions of ICA". Loosely, you could say this means that pixels within each facial feature tend to vary together across faces, and to vary independently from those in other facial features. For this to be true, the face images would probably have to be aligned such that the same set of pixels consistently covers the same facial features across images.
Also, is the primary difference in PCA and ICA that PCA aims to maximize the variance of it's projections which is given by the directions provided by the eigendecomposition of its covariance matrix and ICA aims to maximize the independence of its feature vectors?
Your statement about PCA is correct. I'm not sure I understand the precise intended meaning of your statement about ICA, so I'll phrase it another way: ICA tries to maximize the independence of the projections of the data onto each weight/basis vector. Note that this is distinct from independence of the basis vectors themselves. ICA typically assumes that the data are i.i.d., that there's no noise, and that the number of components is equal to the number of input dimensions. PCA doesn't share the last assumption.
But since the the covariance matrix is a symmetrical matrix, aren't all the eigenvectors independent anyway?
If the data has $d$ dimensions and the covariance matrix has full rank, there will be $d$ linearly independent eigenvectors. A set of vectors is linearly independent if it's impossible to write one of the vectors as a linear combination of the others. This is true of the eigenvectors because they're orthogonal. Note that linear independence is a distinct concept from statistical independence, and linear independence of the basis vectors has nothing to do with ICA. Unlike PCA, ICA doesn't require the basis vectors to be orthogonal. Furthermore, it tries to maximize statistical independence of the projections, not the basis vectors.
