3
$\begingroup$

I am trying to classify two types of objects, which unfortunately have high-dimensional features with few samples. (230 features 12 samples from each group).

As a first step: To reduce the dimension, I have tested three different approaches of PCA (each with slightly different parameters) and use the scores of the first 3 PCs as the features instead of the original 230 features.

As a second step: in order to classify those objects, I have tested three different classifiers (SVM, 3-NN, Naive Bayes) where I use the leave one out method, i.e.: training the classifier on 23 objects and tested it on the one leave out, for testing.

Sum up: I have use 3 different dimensional reduction methods and three use different classifiers (nine combinations overall).

In one combination (out of 9), I got excellent classification performance.

My question: Is there some statistic approach I can use here in order to prove that this option was indeed robust and the correct one, and didn't happen by chance?

$\endgroup$
10
  • $\begingroup$ + please forgive me that I am not a native English speaker :) $\endgroup$
    – Dov
    Oct 26, 2011 at 8:51
  • $\begingroup$ You have only 24 objects and reduced the set to 3 attributes? This is quite harsh. $\endgroup$
    – user88
    Oct 26, 2011 at 9:55
  • 1
    $\begingroup$ BTW, I'm not sure Fourier will work well on EEG signals. wavelet might do better. google.com/… $\endgroup$
    – Dov
    Oct 26, 2011 at 21:28
  • 1
    $\begingroup$ It is quite obvious -- PCA does not depend on decision. Yet I'll try to find some reference. About selecting descriptors, this was a general remark -- but I'm sure you'll easily find what is suitable for EEG. You can even try asking here. $\endgroup$
    – user88
    Oct 26, 2011 at 22:08
  • 1
    $\begingroup$ Comment1: 12 samples (patients) is a very small sample size. Besides the question whether you can train a good model on so few patients (which will depend on the between class : within class variance) you won't be able to show that the classifier works well: if you get 11 of your 12 patients in that group correctly classified, that is a sensitivity of 92%, but the 95% confidence interval goes from 62% - 100%. How do you pick the best of those nine models which have all comparable uncertainty on their measured performance? $\endgroup$ Mar 26, 2012 at 13:33

1 Answer 1

2
$\begingroup$

First of all make sure that classification is really the goal (vs. prediction), e.g., you don't care if a predictor combination that yields a probability of 0.5 is arbitrarily put into one of the two classes. Second, obtain an adequate sample size. Third, statistical principles often tell you that one prediction method is expected to be nearly optimal. Question the need for multiple methods.

$\endgroup$
7
  • $\begingroup$ First, highly appreciated your comment! Searching the web for the definition of classification and prediction: (admit that I wasn't aware of this different) I have found that classification predicts categorical class labels while prediction is used for Models continuous-valued functions. If the above definition is correct, I'm talking about classification here. $\endgroup$
    – Dov
    Oct 26, 2011 at 19:21
  • $\begingroup$ Can you further elaborate when you say "you don't care if a predictor combination that yields a probability of 0.5 is arbitrarily put into one of the two classes" and also on ", statistical principles often tell you that one prediction method is expected to be nearly optimal. Question the need for multiple methods." thanks in advance! $\endgroup$
    – Dov
    Oct 26, 2011 at 19:22
  • 1
    $\begingroup$ Prediction doesn't refer to the nature of the dependent variable per se. It refers to having a continuous result for the predictions, e.g., a predicted probability that someone will be in class B given they are in either class A or class B. An excellent probability modeling method (e.g., logistic model with appropriate nonlinearities and shrinkage) is a good choice. The goal of the analyst, most of the time, should be to give optimal predictions. The decision maker should decide what to do with the predictions. E.g., a probability of 0.5 may indicate "get more data". $\endgroup$ Oct 26, 2011 at 19:55
  • $\begingroup$ thanks again for your time! I have a discrete result for the predictions. -- so, I guess I'm in the classification case.. $\endgroup$
    – Dov
    Oct 26, 2011 at 20:18
  • $\begingroup$ "probability of 0.5 may indicate get more data" --> but what can I do if with one model I'm having a probability of 0.5 and with slightly different model I got a probability of 0.9 $\endgroup$
    – Dov
    Oct 26, 2011 at 20:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.