I'm trying to classify (LDA) few samples (n=12) in a high dimensional feature space (p=24) into 3 classes.

First I reduced the dimension of my initial dataset with a PCA, keeping only the first two Eigen vectors. Update: turns out, I was actually using all 11 PCs for the LDA. Then I had a look at the projection of my n x 2 n x 11 dataset in the LDA space (1st vs 2nd Eigen vector) and I obtained the following: Dataset projected in LDA space

I was quite happy because the LDA found a strong separation between the 3 classes.

So I tried a leave-one-out cross validation to evaluate the LDA. I trained the classifier with 11 samples and tested it with the last one, and looped around.

The problem is the classifier performs at chance level (30% success rate).

I noticed that the LDA space changes drastically between each iteration, depending on the 11 samples used to compute it. Moreover, when I project the tested sample in the corresponding LDA space, it falls quite far away from what should be its group, explaining the poor success rate. LDA space computed with 11 samples, and projection of the last one

My questions are: is it normal that such a (visually) nice separation between classes leads to such a poor classification? Is it due to the small number of samples? Is there anything I can do to improve the situation?

  • $\begingroup$ LDA cannot work at all or work correctly in complete collinearity situation, such as when n<p (12<24 in your case). Most programs will either issue en error or will drop the "surplus" variables as "failing to pass tolerance test". So, don't dream of LDA and especially of any cross-validation until you get much more cases ("samples", in your words). $\endgroup$ – ttnphns Jun 2 '16 at 17:47
  • $\begingroup$ Thank you for your answer! I tried to reduce p with a PCA, training the classifier only with the first two Eigen vectors. I still have a visually nice separation between classes, but the success rate remains at chance level. Is increasing N the only way out here? $\endgroup$ – Khanigh Jun 2 '16 at 18:04
  • $\begingroup$ Welcome to CV and congratulations with such a nicely written first post. +1. One thing is not clear to me: LDA computations require inverting within-class covariance matrix which in your $n<p$ case is singular and hence cannot be inverted. So applying LDA "as is" without any tricks is simply not possible. There are some tricks, but you don't mention any. What program of function are you using to compute LDA? Do you know what is actually being computed here? $\endgroup$ – amoeba Jun 2 '16 at 20:19
  • 1
    $\begingroup$ @amoeba Your comment made me realize that I wasn't feeding my LDA with only PC1 and PC2, but with the whole set of 11 PCs... Hence I think I am in the situation described in your answer to this thread, i.e. over-fitting: near-perfect class separation on the training data with chance performance on the test data. Am I right? Anyway thank you very much for your help! $\endgroup$ – Khanigh Jun 3 '16 at 10:59
  • 1
    $\begingroup$ @amoeba Yes, I don't think my problem is different enough from the thread I mentioned to justify a distinct post. We can close it. $\endgroup$ – Khanigh Jun 3 '16 at 12:02