6
$\begingroup$

I'm playing with support vector machines (SVM) using the e1071::svm() function in R, and I encountered a scenario where I asked it for a leave-one-out cross-validated classification of a 2-category response and obtained a total accuracy of 38% (35/90), which, given 90 samples, ends up with a 95% confidence interval that is below chance. Should I consider this a fluke, and if not, how is it possible for an SVM to become anti-predictive?

In case it matters, I used default values for the cost and gamma parameters, and the data predicting the response was a 8192 item vector representing 500 milliseconds of electroencephalogram data collected across 64 electrodes.

$\endgroup$
10
$\begingroup$

It is very probably the settings for the hyper-parameters that are the issue, leading to severe over-fitting of the data. Without proper tuning of the hyper-parameters, and SVM can perform arbitrarily badly, especially for high dimensional data (it is the tuning of the regularisation parameter that gives robustness against over-fitting in high dimensional spaces). I would suggest nested cross-validation, with the outer (leave-oue-out) cross-validation used form performance estimation and the hyper-parameters tuned independently in each fold by minimising a cross-validation based model selection criterion (I use Nelder-Mead simplex method rather than grid search).

The short answer, is never use default hyper-parameter values, always tune them afresh for each new (partition of the) dataset.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.