Testing if low-variance components in PCA contain any "signal" My problem is similar to this one but I am looking for a different solution: (so if it should be merged just let me know).
Measuring what's 'lost' in PCA dimensionality reduction?
I my application we have a correlation matrix of dimension 30 upon which we conduct a PCA analysis and retain the first three eigenvectors on the basis that they typically contain 90+% of the variation. 
However this has always struck me as a little arbitrary, I would like to test whether these smaller eigenvectors do actually contain a "signal" rather than white noise.
I suppose one very simple method would be to split the data up and see if these smaller eignevectors maintain a similar shape, but I would like to find a more scientifically robust way to test this hypothesis.
 A: Your question is not really possible to answer unless you have additional information about the situation you are applying this to.
Indistinguishable situations
For the purposes of this, we'll assume that $X$, $Y$, and $Z$ are 0-mean multivariate normal distributions in $\mathbb{R}^d$, and we're interested in one or more spectrum $\sigma_i$ (a vector of size $d$ with decreasing values, yada yada). I refer to the components of the spectrum as eigenvalues, without specifying that they're the eigenvalues of the covariance matrix.

*

*The true distribution is $X$ which has spectrum $\sigma_X$ with all non-zero values. There is no error, and we draw a large number of samples, estimating everything very accurately. Clearly all of the "small" eigenvalues still have "information" and aren't noise.


*The true distribution  is $Y$ which has a spectrum $\sigma_Y$ with only 3 non-zero eigenvalues. There's noise, though, so we measure $Y+Z$, where $\sigma_Z$ does have all non-zero eigenvalues. Let's suppose $Y$ and $Z$ are such that $\sigma_{Y+Z} = \sigma_X$. Here, it's obvious that all but the top 3 eigenvalues are "merely noise".
My point is just that which parts of the spectrum can be attributed to "noise" is not a property of the sample.
External criteria
There potentially are external criteria that can help you distinguish the above situations, but they're sort of problem specific. For instance, in the Netflix Challenge, a very successful technique for predicting movie ratings was based on SVD (which is also the basis of PCA). When using SVD-based algorithms for a prediction task, one is confronted with the same challenge you have: "How many non-zero components do I consider? How far do I reduce the dimensionality?" The answer is basically cross validation. The more components you consider, the lower your training error is, but the more risk of overfitting. The validation error is a proxy for generalization error. So, you generally get a chart like:

If you're not doing a predictive problem, I don't really have useful advice, but I do imagine there might be something you want to measure that can help you define what it means for something to be "signal" vs "noise" in your application.
A: The basic way to approach this is to plot a metric of choice versus the amount of eigenvectors kept. I personally prefer to use the 'loss' after reconstruction for my applications, which usually reveals exponentially decreasing rewards.
As an alternative you could also look at the spectral decay. For this, you plot the index of the eigenvector on the x-axis and the related eigenvalue on the y-axis. For PCA, this is sometimes done in a 'cumulative' way to show the cumulative sum of variance explained like so:

