How do I select the number of components for independent components analysis?

Question

In the absence of good a priori guesses about the number of components to request in Independent Components Analysis, I'm looking to automate a selection process. I think that a reasonable criterion might be the number that minimizes the global evidence for correlation amongst the computed components. Here's pseudocode of this approach:

for each candidate number of components, n:
    run ICA specifying n as requested number of components
    for each pair (c1,c2) of resulting components:
        compute a model, m1: lm(c1 ~ 1)
        compute a model, m2: lm(c1 ~ c2)
        compute log likelihood ratio ( AIC(m2)-AIC(m1) ) representing the relative likelihood of a correlation between c1 & c2
    compute mean log likelihood ratio across pairs
Choose the final number of components as that which minimizes the mean log likelihood of component relatedness

I figure this should automatically penalize candidates larger than the "true" number of components because ICAs resulting from such candidates should be forced to distribute information from single true components across multiple estimated components, increasing the average evidence of correlation across pairs of components.

Does this make sense? If so, is there a faster way of achieving an aggregate metric of relatedness across estimated components than the mean log likelihood approach suggested above (which can be rather slow computationally)? If this approach doesn't make sense, what might a good alternative procedure look like?

Answer 1

The Variational Ensemble Learning approach to ICA can do this "automatic relevance detection". It automatically turns off components that are not required to improve the bound on the Bayesian Evidence.

Have a look at the thesis of James Miskin available here, which introduces the technique.

This is implemented very eligently in Java by John Winn (another PhD that implements Bayesian Ensemble Learning via a message passing Algorithm).

To learn the technique, I decided to implement Winn's algorithm in c++ which can be obtained from here (active development).

Answer 2

As Tom says, Automatic Relevance Determination is a good approach to select a small subset of components in a probabilistic model.

Another approach for ICA is to use an Indian Buffet Process prior - Knowles and Ghahramani do this in "Infinite Sparse Factor Analysis and Infinite Independent Components Analysis."