29
$\begingroup$

I've come across a scenario where I have 10 signals/person for 10 people (so 100 samples) containing 14000 data points (dimensions) that I need to pass to a classifier. I would like to reduce the dimensionality of this data and PCA seems to be the way to do so. However, I've only been able to find examples of PCA where the number of samples is greater than the number of dimensions. I'm using a PCA application that finds the PCs using SVD. When I pass it my 100x14000 dataset there are 101 PCs returned so the vast majority of dimensions are obviously ignored. The program indicates the first 6 PCs contain 90% of the variance.

Is it a reasonable assumption that these 101 PCs contain essentially all the variance and the remaining dimensions are neglectable?

One of the papers I've read claims that, using a similar (though slightly lower quality) dataset than my own, they were able to reduce 4500 dimensions down to 80 retaining 96% of the original information. The paper hand-waves over the details of the PCA technique used, only 3100 samples were available, and I have reason to believe less samples than that were used to actually perform PCA (to remove bias from the classification phase).

Am I missing something or is this really the way that PCA is used with high dimensionality-low sample size dataset? Any feedback would be greatly appreciated.

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ If you don't have a lot more data than dimensions it is hard to find a direction that removes most of the variability whihc is what the first principal component is suppose to do. In general there is the curse of dimensionality. Data tends to move away from the center in high dimensions. Bellman wrote about this in the 1960s. $\endgroup$
    – Michael R. Chernick
    May 22, 2012 at 3:30
  • $\begingroup$ Very much related: stats.stackexchange.com/questions/123318. $\endgroup$
    – amoeba
    Apr 9, 2017 at 19:49
  • $\begingroup$ Data-points and dimensions are not the same. A single data-point means a single example. And, the values of all your 14000 dimensions of a single example is basically defining the position of that example point in a 14000 dimensional space. And you have 100 data-points like this in different positions of that 14000 dimensional space as your data-set shape is 100x14000. $\endgroup$
    – hafiz031
    Nov 4, 2020 at 19:55

3 Answers 3

Reset to default
12
$\begingroup$

I'd look at the problem from a slightly different angle: how complex a model can you afford with only 10 subjects / 100 samples?

And that question I usually answer with: much less than 100 PCs. Note that I work on a different type of data (vibrational spectra), so things may vary a bit. In my field a common set up would be using 10 or 25 or 50 PCs calculated from O (1000) spectra of O (10) subjects.

Here's what I'd do:

  • Look at the variance covered by those 100 PCs. I usually find that only few components really contribute to the variance in our data.

  • I very much prefer PLS as pre-treatment for clasification over PCA as it does a much better job at sorting out directions that have a high variation which does not help the classification (in my case that could be focus variations, differing sample thickness, ...). In my experience, I often get similar classifiers with 10 PLS latent variables or 25 to 50 PCs.

  • Validation samples need to be processed with the PCA rotation calculated from the training set only, otherwise the validation can (and in such extreme cases as yours most probably will) have a large overoptimistic bias.
    In other words, if you do out-of-bootstrap or cross validation, the PCA or PLS preprocessing needs to be calculated for each train/test set combination separately.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for the very informative posts. I'm working with biometric signal data. To perform classification with reasonable performance I need less than 100 PCs, somewhere in the range of 25-50 would be fine. I've considered cutting back on my sampling rate to reduce the number of samples I need to examine, but would you expect the resolution trade-off to be worth it or will it have any effect at all given the training same size remains the same? Although I need PCA to be consistent with other studies in the field, I will definitely look into PLS as a secondary approach. $\endgroup$
    – James
    May 23, 2012 at 2:29
  • 1
    $\begingroup$ @James: I'm afraid the sampling rate question cannot be answered without knowing your data. Essentially it is the question of effective sample size. Without knowing anything further, we can only say that it is somewhere between n (persons) and n (samples). If all 10 samples of 1 person are much more similar to each other than to samples of a different person, then more samples do not add much information to the data set. You can check that by building one model with all samples and a second model with only one sample per person and comparing their performance. $\endgroup$
    – cbeleites unhappy with SX
    May 23, 2012 at 13:11
  • $\begingroup$ @James: (part 2) If you think that the samples of one person are rather similar to each other, then you should take care that validation and training sets do not share personse (i.e. all samples of one person are either in training or in test set). $\endgroup$
    – cbeleites unhappy with SX
    May 23, 2012 at 13:13
  • $\begingroup$ Thanks for getting back to me. My data has a pretty high degree of variability with significant overlap in sample space amongst user classes. But sorry, rather than reduce the samples I meant reduce the resolution on the 14000 data points, so say only every 2nd, 3rd or 4th point were used in PCA, but with the same number of actual user samples. I was curious whether such a drop in resolution would be expected to have a positive, negative or no impact at all on the results of the PCA given my dimensions are already much larger than my samples. $\endgroup$
    – James
    May 24, 2012 at 2:30
  • 1
    $\begingroup$ @James, this again depends. But instead of using only every n$^{th}$ point, I'd recommend averaging/binning every n points, so you reduce the dimensionality and increase the signal to noise ratio. Finding out what (whatever your data point dimension is; I'd put "spectral" here) resolution you need is IMHO one point of the characterization of your problem/data/classifier characterization. $\endgroup$
    – cbeleites unhappy with SX
    May 24, 2012 at 16:40
12
$\begingroup$

If $n$ is the number of points and $p$ is the number of dimensions and $n \leq p$ then the number of principal components with non-zero variance cannot exceed $n$ (when doing PCA on raw data) or $n-1$ (when doing PCA on centered data - as usual).

Improve this answer
$\endgroup$
3
6
$\begingroup$

Coming at this from a different angle:

In PCA, you're approximating the covariance matrix by a $k$-rank approximation (that is, you only keep the top $k$ principal components). If you want to picture this, the covariance vectors are being projected orthogonally down into a lower dimensional linear subspace. Since you've only got 100 data points, the sample covariance necessarily lies on a subspace of dimension $\leq$ 100 (actually as ttnphns proves, 99).

Of course, the point is to keep the large PCs and throw out the small ones to avoid fitting noise. You said 6 accounts for 96% of the variance, so that sounds good. Another technique would be to do cross validation and figure out how high $k$ gets before error on the hold-out data increases.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.