5
$\begingroup$

I want to reduce the dimensionality of my data with PCA, until it preserves $\alpha = 0.99$ of the variance. How do I decide how many eigenvectors I should use?

So I'm looking for a function $f(data, \alpha) = k$ in order to use the first $k$ components of the projected data.

$\endgroup$
1
  • $\begingroup$ I'd personally like to see an answer that doesn't rely on ex-post summing of singular values, i.e. brute force. $\endgroup$ Jan 10, 2015 at 15:36

2 Answers 2

4
$\begingroup$

Suppose $X$ is your data matrix. Using SVD you can find all its singular values $s_i$ and then calculate the cumulative sum of squared singular values. When the cumulated sum reaches $99\%$ of total sum, then you have the number, which means selecting the first $k$ PCs, you can preserve $99\%$ variance: $$\sum_{i=1}^{k} s_i^2 = 0.99 \sum s_i^2.$$

Matlab code:

X = bsxfun(@minus, X, mean(X));             %// center the data
[~,s,~] = svd(X);                           %// run SVD
normsqS = sum(s.^2);                        %// total variance
k = find(cumsum(s.^2)/normsqS >= 0.99, 1);  %// find k
$\endgroup$
5
  • $\begingroup$ Thank you! This looks like the correct answer but I would really appreciate a MATLAB example. $\endgroup$
    – jeff
    Jan 10, 2015 at 4:15
  • $\begingroup$ I think this page has a really close code, but I still can't get it : atmos.washington.edu/~breth/classes/AM582/matlab/html/… $\endgroup$
    – jeff
    Jan 10, 2015 at 4:23
  • $\begingroup$ In the page you found, after normsqS = sum(s.^2); run find(cumsum(s.^2)/normsqS>=0.99, 1) $\endgroup$ Jan 10, 2015 at 4:33
  • 1
    $\begingroup$ +1 and I edited your answer to provide a formula and a code, hope you don't mind. $\endgroup$
    – amoeba
    Jan 10, 2015 at 15:04
  • $\begingroup$ @amoeba, I don't mind at all, and thanks for your editation. I am still learning how to write formula and codes in the site. $\endgroup$ Jan 11, 2015 at 2:56
2
$\begingroup$

You can reduce however many principal components you want...as long as you add a diagonal matrix to the k-reduced estimate you can match the variances exactly. For example, where $\mathbf{\widehat\Sigma}_{MOM}$ is the sample covariance matrix, we can write it as $$ \mathbf{\widehat\Sigma}_{MOM} = \mathbf{V\Lambda V^T} = \sum_{i=1}^{p} \lambda_i\mathbf{v}_i\mathbf{v}_i^T $$

Where $T$ is the transpose operator, $p$ the number of variables,$\mathbf{\Lambda}$ the diagonal matrix of eigen values,$\mathbf{V}$ a matrix of column eigen vectors and $\mathbf{v}_i$ the $i^{th}$ eigen vector.

Suppose I choose the first $k<p$ principal components to be significant then I can form a new estimate $$ \mathbf{\widehat\Sigma}_{NEW} = \sum_{i=1}^{k} \lambda_i\mathbf{v}_i \mathbf{v}_i^T+\mathbf{\Psi} $$ where $\mathbf{\Psi}$ is a diagonal matrix constructed such that

$$ \mathrm{diag}(\mathbf{\widehat\Sigma}_{NEW}) = \mathrm{diag}(\mathbf{\widehat\Sigma}_{MOM}) $$ This is a very common approach in many high dimensional problems and can be thought of as a "shrinkage" technique. The problem with the answer given above is that it's a biased/inconsistent estimate of the variance (it will always underestimate the variance). This shrinkage technique will not suffer from such bias and allows you to specify however many principal components you want to keep. This comes in super handy when the rank of the covariance matrix is small.

There are a lot of different methods for choosing $k$, this Wikipedia page highlights a few simpler ones. Personally I use Monte Carlo simulations predicated on Parallel Analysis. The method I use is robust to extreme rank deficiency, is reasonably quick, and predicts the correct number of significant principal components around 85% of the time at the 0.05 rejection level (and is usually pretty close when not exactly correct).

If you want to be really quick you can use Random matrix Theory (RMT) techniques described here and here . The last reference also includes interesting Bayesian techniques if your into that sort of thing. RMT has a closed form solution, but it's based on asymptotics which may or may not hold up well with fewer observations or semi-definite covariance matrices.

$\endgroup$
3
  • 2
    $\begingroup$ What would the point of keeping the total variance exactly the same (by adding diagonal matrix)? Also, I think OP wants to have principal components themselves (i.e. projections of the data on the eigenvectors $v$). Would you then scale them up, so that their summed variance matches the original total variance? If so, what for? $\endgroup$
    – amoeba
    Jan 10, 2015 at 15:07
  • $\begingroup$ @amoeba the OP wished to reduce the dimensionality of the data while preserving 99% of the variance. I wanted to show that by using an alternative estimate of the covariance matrix (one that is well supported in the literature), it was possible to reduce the dimensionality of the covariance matrix while maintaining the unbiased estimate of variance (preserving 100% of the variance), and in such a way that gave you freedom to select any $k$ you wanted. $\endgroup$ Jan 10, 2015 at 21:21
  • $\begingroup$ @amoeba For purposes of estimation and inference, it often turns out that a new covariance matrix is all you need, so just changing the covariance in effect accomplishes the dimensionality reduction, without having to change the data. Additionally in problems where dimensionality reduction is actually desired, being required to maintain 99% of the original variance is often too restrictive and will be ineffective. $\endgroup$ Jan 10, 2015 at 21:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.