0
$\begingroup$

I have gone through the following 2 steps to create a 4-class SVM model:

  1. 5 fold cross validation with grid search to find a C and Gamma value with the lowest error.

  2. Train a model with my complete data set using the C and Gamma values found above.

And now for the 3rd step, I am not sure what the best way is to report the predictive accuracy of my model. One thing which confuses me is the 'Cross Validation accuracy (90%)' reported in Step 1, what is this and I can't report this as my classification accuracy right?

The two metrics I am considering reporting are True detection (Accuracy), and a confusion matrix.

My data consists of four classes, each with only 10 samples, and each sample has 264 features.

I know it is very little data, I could possibly measure another 20 samples for each class over two days, but would rather explore how far I can take my current set first.

$\endgroup$
1
$\begingroup$

I'm afraid you won't get anywhere with 10 cases per class. Even if your classes are so separated that you could classify data by "naked" eye.

First of all, if you want to auto-tune parameters like C and γ, you need a nested validation setup, e.g. nested cross validation. That is, an outer cross validation loop around your present setup which keeps test cases that are independent of the ones for training and tuning (which is actually rather a part of training).

But also consider the random uncertainty you have with so few cases. 90 % accuracy for 40 cases translates for a 95% binomial confidence interval ranging from 76 - 97 % accuracy. How is your auto-tuning to work with that uncertainty?
=> Some more days in the lab are indicated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.