PCA when the dimensionality is greater than the number of samples I've come across a scenario where I have 10 signals/person for 10 people (so 100 samples) containing 14000 data points (dimensions) that I need to pass to a classifier. I would like to reduce the dimensionality of this data and PCA seems to be the way to do so. However, I've only been able to find examples of PCA where the number of samples is greater than the number of dimensions. I'm using a PCA application that finds the PCs using SVD. When I pass it my 100x14000 dataset there are 101 PCs returned so the vast majority of dimensions are obviously ignored. The program indicates the first 6 PCs contain 90% of the variance. 
Is it a reasonable assumption that these 101 PCs contain essentially all the variance and the remaining dimensions are neglectable?
One of the papers I've read claims that, using a similar (though slightly lower quality) dataset than my own, they were able to reduce 4500 dimensions down to 80 retaining 96% of the original information. The paper hand-waves over the details of the PCA technique used, only 3100 samples were available, and I have reason to believe less samples than that were used to actually perform PCA (to remove bias from the classification phase). 
Am I missing something or is this really the way that PCA is used with high dimensionality-low sample size dataset? Any feedback would be greatly appreciated.
 A: Coming at this from a different angle:
In PCA, you're approximating the covariance matrix by a $k$-rank approximation (that is, you only keep the top $k$ principal components). If you want to picture this, the covariance vectors are being projected orthogonally down into a lower dimensional linear subspace. Since you've only got 100 data points, the sample covariance necessarily lies on a subspace of dimension $\leq$ 100 (actually as ttnphns proves, 99).
Of course, the point is to keep the large PCs and throw out the small ones to avoid fitting noise. You said 6 accounts for 96% of the variance, so that sounds good. Another technique would be to do cross validation and figure out how high $k$ gets before error on the hold-out data increases.
A: If $n$ is the number of points and $p$ is the number of dimensions and $n \leq p$ then the number of principal components with non-zero variance cannot exceed $n$ (when doing PCA on raw data) or $n-1$ (when doing PCA on centered data - as usual).
A: I'd look at the problem from a slightly different angle: how complex a model can you afford with only 10 subjects / 100 samples?
And that question I usually answer with: much less than 100 PCs. Note that I work on a different type of data (vibrational spectra), so things may vary a bit. In my field a common set up would be using 10 or 25 or 50 PCs calculated from O (1000) spectra of O (10) subjects.
Here's what I'd do:


*

*Look at the variance covered by those 100 PCs. I usually find that only few components really contribute to the variance in our data.

*I very much prefer PLS as pre-treatment for clasification over PCA as it does a much better job at sorting out directions that have a high variation which does not help the classification (in my case that could be focus variations, differing sample thickness, ...). In my experience, I often get similar classifiers with 10 PLS latent variables or 25 to 50 PCs.

*Validation samples need to be processed with the PCA rotation calculated from the training set only, otherwise the validation can (and in such extreme cases as yours most probably will) have a large overoptimistic bias.
In other words, if you do out-of-bootstrap or cross validation, the PCA or PLS preprocessing needs to be calculated for each train/test set combination separately.
