Can anyone explain why I have obtained an anti-predictive Support Vector Machine?

Question

I'm playing with support vector machines (SVM) using the e1071::svm() function in R, and I encountered a scenario where I asked it for a leave-one-out cross-validated classification of a 2-category response and obtained a total accuracy of 38% (35/90), which, given 90 samples, ends up with a 95% confidence interval that is below chance. Should I consider this a fluke, and if not, how is it possible for an SVM to become anti-predictive?

In case it matters, I used default values for the cost and gamma parameters, and the data predicting the response was a 8192 item vector representing 500 milliseconds of electroencephalogram data collected across 64 electrodes.

Answer 1

It is very probably the settings for the hyper-parameters that are the issue, leading to severe over-fitting of the data. Without proper tuning of the hyper-parameters, and SVM can perform arbitrarily badly, especially for high dimensional data (it is the tuning of the regularisation parameter that gives robustness against over-fitting in high dimensional spaces). I would suggest nested cross-validation, with the outer (leave-oue-out) cross-validation used form performance estimation and the hyper-parameters tuned independently in each fold by minimising a cross-validation based model selection criterion (I use Nelder-Mead simplex method rather than grid search).

The short answer, is never use default hyper-parameter values, always tune them afresh for each new (partition of the) dataset.