Cross validating one-against-all SVMs

Question

I'm training a bunch of SVM models to do one-against-all multiclass classification (a test instance is classified as the class which produces the largest positive SVM response).

What's the best way to do cross-validation for selection of the regularization parameter?

Should I do cross-validation separately for each SVM model I train, potentially getting a different regularization parameter for each of the models? Or should I cross-validate as a group, where I try a particular regularization parameter across all models?

Also, what is a good metric for cross-validation? Accuracy? Precision? F-Measure?

Answer 1

Since you're treating your SVMs as an ensemble, you should cross validate them as an ensemble. This could potential mean cross-validating multiple combinations of regularization parameters and assessing out-of-sample accuracy of the combined ensemble. Lets say you have 5 SVMs and you want to test 5 possible regularization parameters. This means you have to cross-validate 25 ensembles, one for each possible parameter combination. For each cross validation compute the accuracy (or precision, or whatever) for the entire ensemble on the multi-class problem.

The metric you use is subjective and depends on the issues at hand. Accuracy is a good place to start.

Going a little further, many SVM implementations support multi-class problems out of the box. Why not just use one of those, and reduce your problem to cross-validating one SVM?