14
$\begingroup$

I want to use the Akaike Information Criterion (AIC) to choose the appropriate number of factors to extract in a PCA. The only issue is that I'm not sure how to determine the number of parameters.

Consider a $T\times N$ matrix $X$, where $N$ represents the number of variables and $T$ the number of observations, such that $X\sim \mathcal N\left(0,\Sigma\right)$. Since the covariance matrix is symmetric, then a maximum likelihood estimate of $\Sigma$ could set the number of parameters in the AIC equal to $\frac{N\left(N+1\right)}{2}$.

Alternatively, in a PCA, you could extract the first $f$ eigenvectors and eigenvalues of $\Sigma$, call them $\beta_{f}$ and $\Lambda_{f}$ and then calculate $$\Sigma=\beta_{f}\Lambda_{f}\beta_{f}'+I\sigma_{r}^{2}$$ where $\sigma_{r}^{2}$ is the average residual variance. By my count, if you have $f$ factors, then you would $f$ parameters in $\Lambda_{f}$, $Nf$ parameters in $\beta_{f}$, and $1$ parameter in $\sigma_{r}^{2}$.

Is this approach correct? It seems like it would lead to more parameters than the maximum likelihood approach as the number of factors increases to $N$.

Improve this question
$\endgroup$
9
  • 2
    $\begingroup$ $N f$ overcounts the parameters: there is redundancy due to the fact that the eigenvectors are mutually orthogonal. $\endgroup$
    – whuber ♦
    Aug 21, 2012 at 17:12
  • 11
    $\begingroup$ The first eigenvector has $N$ free parameters. The orthogonality condition restricts the second eigenvector to the hyperspace orthogonal to the first, needing only $N-1$ parameters. Each successive eigenvector needs one less parameter than the preceding one did. At the limit of $N$ eigenvectors you discard $\sigma_r^2$ (because it's now zero), giving $N+(N-1)+\cdots+1$ = $N(N+1)/2$ parameters in toto, in agreement with your first parameter count. $\endgroup$
    – whuber ♦
    Aug 21, 2012 at 18:04
  • 1
    $\begingroup$ @A.Donda The situation is vague: let's suppose that you have also indicated the multiplicity of each eigenvalue and that these multiplicities are $n_1, n_2, \ldots, n_s,$ summing to $N.$ Allowing that PCA finds an orthogonal transformation, we would have $N(N-1)/2$ parameters to determine it. But the stabilizers of each eigenspace are orthogonal groups in the dimensions $n_i.$ Each thereby eliminates $n_i(n_i-1)/2$ parameters, leaving $$N(N-1)/2 - \sum_{i=1}^s n_i(n_i-1)/2$$ parameters for rotations. The $s$ eigenvalues supply the remaining parameters. $\endgroup$
    – whuber ♦
    Apr 19, 2018 at 16:00
  • 1
    $\begingroup$ (I should add that the application of this counting to the question is dubious: the PCA uses all $N(N-1)/2$ parameters, even if it might happen to find some eigenvalues of higher multiplicity. And in almost any real dataset, it will never obtain a multiplicity greater than $1$ anyway.) $\endgroup$
    – whuber ♦
    Apr 19, 2018 at 16:02
  • 1
    $\begingroup$ @whuber, thanks! My question is motivated by a situation where I estimate a covariance matrix under a constraint on the eigenvalues. $\endgroup$
    – A. Donda
    Apr 19, 2018 at 16:08

4 Answers 4

Reset to default
6
$\begingroup$

The works of Minka (Automatic choice of dimensionality for PCA, 2000) and of Tipping & Bishop (Probabilistic Principal Component Analysis) regarding a probabilistic view of PCA might provide you with the framework you interested in. Minka's work provides an approximation of the log-likelihood $\mathrm{log}\: p(D|k)$ where $k$ is the latent dimensionality of your dataset $D$ by using a Laplace approximation; as stated explicitly : "A simplification of Laplace's method is the BIC approximation."

Clearly this takes a Bayesian viewpoint of your problem that is not based on the information theory criteria (KL-divergence) used by AIC.

Regarding the original "determination of parameters' number" question I also think @whuber's comment carries the correct intuition.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I was playing around with AIC versus AICc on random matrices of varying sizes. AICc seemed to be working better. Those references look good, but I haven't had chance to digest yet. $\endgroup$
    – John
    Aug 22, 2012 at 21:08
6
$\begingroup$

Selecting an "appropriate" number of components in PCA can be performed elegantly with Horn's Parallel Analysis (PA). Papers show that this criterion consistently outperforms rules of thumb such as the elbow criterion or Kaiser's rule. The R package "paran" has an implementation of PA that requires only a couple of mouse clicks.

Of course, how many components you retain depends on the goals of the data reduction. If you only wish to retain variance that is "meaningful", PA will give an optimal reduction. If you wish to minimize the information loss of the original data, however, you should retain enough components to cover 95% explained variance. This will obviously keep many more components than PA, although for high-dimensional datasets, the dimensionality reduction will still be considerable.

One final note about PCA as a "model selection" problem. I don't fully agree with Peter's reply. There have been a number of papers that reformulated PCA as a regression-type problem, such as Sparse PCA, Sparse Probabilistic PCA, or ScotLASS. In these "model-based" PCA solutions, loadings are parameters that can be set to 0 with appropriate penalty terms. Presumably, in this context, it would also be possible to calculate AIC or BIC type statistics for the model under consideration.

This approach could theoretically include a model where, for example, two PCs are unrestricted (all loadings non-zero), versus a model where PC1 is unrestricted and PC2 has all loadings set to 0. This would be equivalent to inferring whether PC2 is redundant on the whole.

References (PA):

  • Dinno, A. (2012). paran: Horn's Test of Principal Components/Factors. R package version 1.5.1. http://CRAN.R-project.org/package=paran
  • Horn J.L. 1965. A rationale and a test for the number of factors in factor analysis. Psychometrika. 30: 179–185
  • Hubbard, R. & Allen S.J. (1987). An empirical comparison of alternative methods for principal component extraction. Journal of Business Research, 15, 173-190.
  • Zwick, W.R. & Velicer, W.F. 1986. Comparison of Five Rules for Determining the Number of Components to Retain. Psychological Bulletin. 99: 432–442
Improve this answer
$\endgroup$
1
  • $\begingroup$ Welcome to the site, @BenM. From your answer, I think it'll be nice having you around (although I don't know enough about PCA beyond the basics to evaluate your claims). One question, you note that these positions have been well established, could you list a couple of representative publications where the interested reader could find more detail? $\endgroup$
    – gung - Reinstate Monica
    Jan 25, 2013 at 20:24
-1
$\begingroup$

AIC is designed for model selection. This is not really a model selection problem and maybe you would be better off taking a different approach. An alternative could be to specify a certain total percentage of variance explained (like say 75%) and stop when the percentage reaches 75% if it ever does.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ I'm selecting between different models based on the number of factors (the model with 1 factor versus the model with 2, etc). The problem with percentage of variance is mainly that it ignores the cost of estimating additional eigenvectors, especially when the number of observations is less than the number of variables. AIC fits in well with a probabilistic PCA approach. $\endgroup$
    – John
    Aug 21, 2012 at 17:35
  • 3
    $\begingroup$ Michael, could you explain precisely why this is not a model selection problem? It looks like John has clearly formulated it as one. $\endgroup$
    – whuber ♦
    Aug 21, 2012 at 21:21
  • $\begingroup$ @whuber What is the statistical model? It seems to me that deciding the number of principal component used to represent the x% of the variance in a variable Y is not selecting a model. I would not think of the principal components as model parameters either. $\endgroup$
    – Michael R. Chernick
    Aug 21, 2012 at 22:01
  • 2
    $\begingroup$ Consider 2D vectors $X_i$ drawn iid from $N(0,\Sigma)$. We can parameterize $\Sigma$ in terms of two variances $\sigma_i^2$ and a correlation $\rho$. Nested within this model would be the model $|\rho|=1$. Now, we can also parameterize it in terms of the angle $\theta$ of the first principal component and the eigenvalues $\lambda_1\ge\lambda_2$ of those components. Nested within this would be the model $\lambda_2=0$. Both perspectives test for perfect correlation (collinearity); they just use different parameterizations. If you allow the first as a model, you must allow the second. $\endgroup$
    – whuber ♦
    Aug 21, 2012 at 22:09
-3
$\begingroup$

AIC is not appropriate here. You are not selecting among models with varying numbers of parameters - a principal component is not a parameter.

There are a number of methods of deciding on the number of factors or components from a factor analysis or principal component analysis - scree test, eigenvalue > 1, etc. But the real test is substantive: What number of factors makes sense? Look at the factors, consider the weights, figure out which is best suited to your data.

Like other things in statistics, this is not something that can easily be automated.

Improve this answer
$\endgroup$
7
  • 4
    $\begingroup$ If "a principal component is not a parameter", then a fortiori no coefficient of $\Sigma$ can be a parameter, either (because $\Sigma$ is completely determined by the principal component decomposition). This is a puzzling assertion. $\endgroup$
    – whuber ♦
    Aug 21, 2012 at 21:20
  • 1
    $\begingroup$ @whuber A parameter of a covariance matrix maybe but not a model parameter. I side with Peter on this. $\endgroup$
    – Michael R. Chernick
    Aug 21, 2012 at 22:03
  • 3
    $\begingroup$ Peter, exactly what distinction are you making between a "model parameter" and a "parameter"? I am unaware of any such thing and so would appreciate learning about this. If your purpose is to find a parsimonious description of the multivariate covariances, don't they constitute "model" parameters? $\endgroup$
    – whuber ♦
    Aug 22, 2012 at 14:36
  • 3
    $\begingroup$ Peter, a lot of work on this has been done under the name of "low-rank models." Applications include spectral analysis of time series, spatial generalizations thereof, and splines. In the time series case, for instance, a sequence of $n$ observations can be summarized by $m\ll n$ Fourier series components, using essentially the same machinery and concepts as PCA: one retains the eigenvectors (i.e., sine and cosine waves) corresponding to the largest eigenvalues (that is, the amplitudes or powers of the waves). $\endgroup$
    – whuber ♦
    Aug 23, 2012 at 12:53
  • 1
    $\begingroup$ Thanks for the info. Time series is one area of statistics I know little about. $\endgroup$
    – Peter Flom
    Aug 23, 2012 at 15:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.