3
$\begingroup$

I've read:

but I still don't get it.

My problem is to construct a simple SVM, tune it's parameters and calculate the generalization error. Assume I have a dataset with e.g. 10 features and 200 samples. I don't want to waste data, since it's relatively small. My approach would be:

  1. Split the dataset (e.g., 70/30) with holdout method into training / validation and test set.

  2. Make a repeated n-fold cross-validation on the training set. I calculate the error rate (misclassification rate) after the folds are done (simply counting all misclassified samples). I try to minimize the error rate (or any other loss function?) each time the complete n fold is done. I store the error rate and choose the parameters with the lowest error rate. Then I retrain the model on the complete training set with the estimated parameters.

  3. Calculate generalization error with the test set.

The problems:

  • The larger my test set is, the smaller gets the train set, so I discard potential information. Can this be solved via a "stacked" n-fold cv?
  • Do I really have to make a REPEATED n-fold cv? Are there other possibilities?
  • Is the error rate an appreciate loss function or should I choose another one (eg. the empirical error function or mse, but then I'd need a probability output, right?)?
$\endgroup$
2
$\begingroup$

The larger my test set is, the smaller gets the train set, so I discard potential information. Can this be solved via a "stacked" n-fold cv?

Yes. It is usually called nested or double cross validation, and we have a number of questions and answers about that. You could start e.g. with Nested cross validation for model selection

Do I really have to make a REPEATED n-fold cv? Are there other possibilities?

Repetitions / iterations in resampling validation help only if the (surrogate) models are unstable. If you are really sure your models are stable (but how can you be when having concerns about small sample size?) then you don't need the iterations / repetitions. OTOH, IMHO the easiest way to prove that the models are stable is running a few iterations and look at the stability of the predictions.

Is the error rate an appreciate loss function or should I choose another one (eg. the empirical error function or MSE, but then I'd need a probability output, right?)?

No, overall error rate is not a very good loss function, particularly not for optimization. MSE is much better, it is a proper scoring rule. Yes, proper scoring rules need probability output.
However, SVM are anyways quite ugly to optimize as they do not react continuously to small continuous changes in the training data + hyperparameters: up to a certain limit nothing changes (i.e. the same cases stay support vectors), then suddenly the support vectors change.

See also

$\endgroup$
  • $\begingroup$ Thank you very much for the given keywords + links! I've read some paper about proper score rules and this one: google.de/… where the 'empirical error' is used. This seems not to be a proper scoring rule. Why does it work? Thank you! $\endgroup$ – Jan Jun 13 '14 at 10:39
  • 1
    $\begingroup$ @user8648: I just had a glance at that paper, 1st thought: empirical error looks very much like MAE. 2nd thought: if you have enough cases, also inefficient measures will eventually work. 3rd thought: Does it actually work? They validate with % error rate and a test set of 2000 cases. The binomial 95% confidence interval for 4.5% errors observed over 2000 test cases is ca. $\pm$ 0.9%. That means they cannot distinguish all presented models are actually quite central within that window, so based on the random uncertainty of their testing results, they cannot distinguish the models. $\endgroup$ – cbeleites Jun 13 '14 at 11:33
  • $\begingroup$ Thanks! That makes sense. I've tried to use the mse loss function but I needed the posteriori probabilities. So I tried to estimate them as mentioned in citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.1639 . But there is a big problem: To estimate the probability I need a well tuned SVM. To tune the SVM I need the prob output. What can be done here? $\endgroup$ – Jan Jun 13 '14 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.