I'm training a bunch of SVM models to do one-against-all multiclass classification (a test instance is classified as the class which produces the largest positive SVM response).

What's the best way to do cross-validation for selection of the regularization parameter?

Should I do cross-validation separately for each SVM model I train, potentially getting a different regularization parameter for each of the models? Or should I cross-validate as a group, where I try a particular regularization parameter across all models?

Also, what is a good metric for cross-validation? Accuracy? Precision? F-Measure?


Since you're treating your SVMs as an ensemble, you should cross validate them as an ensemble. This could potential mean cross-validating multiple combinations of regularization parameters and assessing out-of-sample accuracy of the combined ensemble. Lets say you have 5 SVMs and you want to test 5 possible regularization parameters. This means you have to cross-validate 25 ensembles, one for each possible parameter combination. For each cross validation compute the accuracy (or precision, or whatever) for the entire ensemble on the multi-class problem.

The metric you use is subjective and depends on the issues at hand. Accuracy is a good place to start.

Going a little further, many SVM implementations support multi-class problems out of the box. Why not just use one of those, and reduce your problem to cross-validating one SVM?

  • 2
    $\begingroup$ @Articuno: The e1071 package in R provides an interface to libsvm as well as functions for one-against-all multiclass SVMs based on libsvm. For example model <- svm(Species ~ ., data = iris, cost = 100, gamma = 1) $\endgroup$ – Zach Dec 5 '11 at 19:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy