PCA Model Selection Using AIC

Question

I want to use the Akaike Information Criterion (AIC) to choose the appropriate number of factors to extract in a PCA. The only issue is that I'm not sure how to determine the number of parameters.

Consider a $T\times N$ matrix $X$, where $N$ represents the number of variables and $T$ the number of observations, such that $X\sim N\left(0,\Sigma\right)$. Since the covariance matrix is symmetric, then a maximum likelihood estimate of $\Sigma$ could set the number of parameters in the AIC equal to $\frac{N\left(N+1\right)}{2}$.

Alternately, in a PCA, you could extract the first $f$ eigenvectors and eigenvalues of $\Sigma$, call them $\beta_{f}$ and $\Lambda_{f}$ and then calculate $$\Sigma=\beta_{f}\Lambda_{f}\beta_{f}'+I\sigma_{r}^{2}$$ where $\sigma_{r}^{2}$ is the average residual variance. By my count, if you have $f$ factors, then you would $f$ parameters in $\Lambda_{f}$, $Nf$ parameters in $\beta_{f}$, and $1$ parameter in $\sigma_{r}^{2}$.

Is this approach correct? It seems like it would lead to more parameters than the maximum likelihood approach as the number of factors increases to $N$.

Answer 1

The works of Minka (Automatic choice of dimensionality for PCA) and of Tipping & Bishop (Probabilistic Principal Component Analysis) regarding a probabilistic view of PCA might provide you with the framework you interested in. Minka's work provides an approximation of the loglikelihood $log p(D|k)$ where $k$ is the latent dimensionality of your dataset $D$ by using a Laplace approximation; as stated explicitly : "A simplification of Laplace's method is the BIC approximation." Clearly this takes a Bayesian viewpoint of your problem that is not based on the information theory criteria (KL-divergence) used by AIC. Regarding the original "determination of parameters' number" question I also think whuber's comment carries the correct intuition.

Answer 2

Selecting an "appropriate" number of components in PCA can be performed elegantly with Horn's Parallel Analysis (PA). Papers show that this criterion consistently outperforms rules of thumb such as the elbow criterion or Kaiser's rule. The R package "paran" has an implementation of PA that requires only a couple of mouse clicks.

Of course, how many components you retain depends on the goals of the data reduction. If you only wish to retain variance that is "meaningful", PA will give an optimal reduction. If you wish to minimize the information loss of the original data, however, you should retain enough components to cover 95% explained variance. This will obviously keep many more components than PA, although for high-dimensional datasets, the dimensionality reduction will still be considerable.

One final note about PCA as a "model selection" problem. I don't fully agree with Peter's reply. There have been a number of papers that reformulated PCA as a regression-type problem, such as Sparse PCA, Sparse Probabilistic PCA, or ScotLASS. In these "model-based" PCA solutions, loadings are parameters that can be set to 0 with appropriate penalty terms. Presumably, in this context, it would also be possible to calculate AIC or BIC type statistics for the model under consideration.

This approach could theoretically include a model where, for example, two PCs are unrestricted (all loadings non-zero), versus a model where PC1 is unrestricted and PC2 has all loadings set to 0. This would be equivalent to inferring whether PC2 is redundant on the whole.

References (PA):

• Dinno, A. (2012). paran: Horn's Test of Principal Components/Factors. R package version 1.5.1. http://CRAN.R-project.org/package=paran
• Horn J.L. 1965. A rationale and a test for the number of factors in factor analysis. Psychometrika. 30: 179–185
• Hubbard, R. & Allen S.J. (1987). An empirical comparison of alternative methods for principal component extraction. Journal of Business Research, 15, 173-190.
• Zwick, W.R. & Velicer, W.F. 1986. Comparison of Five Rules for Determining the Number of Components to Retain. Psychological Bulletin. 99: 432–442

Answer 3

AIC is designed for model selection. This is not really a model selection problem and maybe you would be better off taking a different approach. An alternative could be to specify a certain total percentage of variance explained (like say 75%) and stop when the percentage reaches 75% if it ever does.

Answer 4

AIC is not appropriate here. You are not selecting among models with varying numbers of parameters - a principal component is not a parameter.

There are a number of methods of deciding on the number of factors or components from a factor analysis or principal component analysis - scree test, eigenvalue > 1, etc. But the real test is substantive: What number of factors makes sense? Look at the factors, consider the weights, figure out which is best suited to your data.

Like other things in statistics, this is not something that can easily be automated.