My problem is similar to this one but I am looking for a different solution: (so if it should be merged just let me know).

Measuring what's 'lost' in PCA dimensionality reduction?

I my application we have a correlation matrix of dimension 30 upon which we conduct a PCA analysis and retain the first three eigenvectors on the basis that they typically contain 90+% of the variation.

However this has always struck me as a little arbitrary, I would like to test whether these smaller eigenvectors do actually contain a "signal" rather than white noise.

I suppose one very simple method would be to split the data up and see if these smaller eignevectors maintain a similar shape, but I would like to find a more scientifically robust way to test this hypothesis.

  • $\begingroup$ Your intuition seems related to “parallel analysis”. $\endgroup$
    – Gala
    Sep 10 '13 at 13:28

The basic way to approach this is to plot a metric of choice versus the amount of eigenvectors kept. I personally prefer to use the 'loss' after reconstruction for my applications, which usually reveals exponentially decreasing rewards.

As an alternative you could also look at the spectral decay. For this, you plot the index of the eigenvector on the x-axis and the related eigenvalue on the y-axis. For PCA, this is sometimes done in a 'cumulative' way to show the cumulative sum of variance explained like so:

enter image description here

  • 1
    $\begingroup$ Hi Ciri, can you give some examples of the metric's you refer too in the first part? If I understand correctly though the alternative that you show will just illustrate that the small eigenvalues Principal Components add little to the total variation which is a nice graphical way of validating the procedure we use currently. My point is that even if these eigenvalues are small they may still be contributing important information and I would like a way to test for this? $\endgroup$
    – Baz
    Sep 10 '13 at 15:26
  • 1
    $\begingroup$ The variance explained can be a very important indicator for contributed information. Other measures include information gain, entropy, ... For me, it usually boils down to look at the loss function for my particular application. For instance: online.liebertpub.com/doi/pdf/10.1089/big.2013.0037 here is a study where we added more and more features and then inspected the AUC (Fig. 4, domain: classifcation in human behavioral data). You could do the same with the recovered signal by adding more and more components. $\endgroup$
    – ciri
    Oct 25 '13 at 11:04

Your question is not really possible to answer unless you have additional information about the situation you are applying this to.

Indistinguishable situations

For the purposes of this, we'll assume that $X$, $Y$, and $Z$ are 0-mean multivariate normal distributions in $\mathbb{R}^d$, and we're interested in one or more spectrum $\sigma_i$ (a vector of size $d$ with decreasing values, yada yada). I refer to the components of the spectrum as eigenvalues, without specifying that they're the eigenvalues of the covariance matrix.

  1. The true distribution is $X$ which has spectrum $\sigma_X$ with all non-zero values. There is no error, and we draw a large number of samples, estimating everything very accurately. Clearly all of the "small" eigenvalues still have "information" and aren't noise.

  2. The true distribution is $Y$ which has a spectrum $\sigma_Y$ with only 3 non-zero eigenvalues. There's noise, though, so we measure $Y+Z$, where $\sigma_Z$ does have all non-zero eigenvalues. Let's suppose $Y$ and $Z$ are such that $\sigma_{Y+Z} = \sigma_X$. Here, it's obvious that all but the top 3 eigenvalues are "merely noise".

My point is just that which parts of the spectrum can be attributed to "noise" is not a property of the sample.

External criteria

There potentially are external criteria that can help you distinguish the above situations, but they're sort of problem specific. For instance, in the Netflix Challenge, a very successful technique for predicting movie ratings was based on SVD (which is also the basis of PCA). When using SVD-based algorithms for a prediction task, one is confronted with the same challenge you have: "How many non-zero components do I consider? How far do I reduce the dimensionality?" The answer is basically cross validation. The more components you consider, the lower your training error is, but the more risk of overfitting. The validation error is a proxy for generalization error. So, you generally get a chart like:

Training/Validation Error as a function of Model Capacity

If you're not doing a predictive problem, I don't really have useful advice, but I do imagine there might be something you want to measure that can help you define what it means for something to be "signal" vs "noise" in your application.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.