Does ICA require to run PCA first?

Question

I reviewed an application-based paper saying that applying PCA before applying ICA (using fastICA package). My question is, does ICA (fastICA) require PCA to be run first?

This paper mentioned that

... it is also argued that pre-applying PCA enhances ICA performance by (1) discarding small trailing eigenvalues before whitening and (2) reducing computational complexity by minimizing pair-wise dependencies. PCA decorrelates the input data; the remaining higher-order dependencies are separated by ICA.

Also other papers are applying PCA before ICA, e.g., this one.

Are there any other pros and cons to run PCA before ICA? Please provide theory with references.

Answer 1

The fastICA approach does require a pre-whitening step: the data are first transformed using PCA, which leads to a diagonal covariance matrix, and then each dimension is normalized such that the covariance matrix is equal to the identity matrix (whitening).

There are infinite transformations of the data which result in identity covariance matrix, and if your sources were Gaussian you would stop there (for Gaussian multivariate distributions, mean and covariance are sufficient statistics), in the presence of non-Gaussian sources you can minimize some measure of dependence on the whitened data, therefore you look for a rotation of the whitened data that maximizes independence. FastICA achieves this using information theoretic measures and a fixed-point iteration scheme.

I would recommend the work of Hyvärinen to get a deeper understanding of the problem:

• A. Hyvärinen. Fast and Robust Fixed-Point Algorithms for Independent Component Analysis. IEEE Transactions on Neural Networks 10(3):626-634, 1999.
• A. Hyvärinen, J. Karhunen, E. Oja, Independent Component Analysis, Wiley & Sons. 2001

Please note that doing PCA and doing dimension reduction are not exactly the same thing: when you have more observations (per signal) than signals, you can perform a PCA retaining 100% of the explained variance, and then continue with whitening and fixed point iteration to obtain an estimate of the independent components. Whether you should perform dimension reduction or not is highly context dependent and it is based on your modeling assumptions and data distribution.

Answer 2

Applying PCA to your data has the only effect of rotating the original coordinate axes. It is a linear transformation, exactly like for example Fourier transform. Therefore as such it can really not do anything to your data.

However, data represented in the new PCA space has some interesting properties. Following coordinate rotation with PCA, you may discard some dimensions based on established criteria such as percentage of total variance explained by the new axes. Depending on your signal, you may achieve a considerable amount of dimensional reduction by this method and this would definitely increase the performance of the following ICA. Doing an ICA without discarding any of the PCA components will have no impact on the result of the following ICA.

Furthermore, one can also easily whiten the data in the PCA space due to the orthogonality of the coordinate axes. Whitening has the effect of equalizing variances across all dimensions. I would argue that this is necessary for an ICA to work properly. Otherwise only few PCA components with largest variances would dominate ICA results.

I don't really see any drawbacks for PCA based preprocessing before an ICA.

Giancarlo cites already the best reference for ICA...

Answer 3

The derivation of the fastICA algorithm only requires whitening for a single step. First, you pick the direction of the step (like a gradient descent) and this does not require whitened data. Then, we have to pick the step size, which depends on the inverse of the Hessian. If the data is whitened then this Hessian is diagonal and invertible.

So is it required? If you just fixed the step size to a constant (therefore not requiring whitening) you would have standard gradient descent. Gradient descent with a fixed small step size will typically converge, but possibly much slower than the original method. On the other hand, if you have a large data matrix then the whitening could be quite expensive. You might be better off even with the slower convergence you get without whitening.

I was surprised to not see mention of this in any literature. One paper discusses the problem: New Fast-ICA Algorithms for Blind Source Separation without Prewhitening by Jimin Ye and Ting Huang.

They suggest a somewhat cheaper option to whitening. I wish they had included the obvious comparison of just running ICA without whitening as a baseline, but they didn't. As one further data point I have tried running fastICA without whitening on toy problems and it worked fine.

Update: another nice reference addressing whitening is here: robust independent component analysis, Zaroso and Comon. They provide algorithms that do not require whitening.