Linear SVM C optimisation data, how to partition into train, model construction, test

Question

My aim is to find the best C for a linear SVM classification using libsvm, I have 120 instances in total and 2 classes which I want to classify.

I have question regarding partitioning a the dataset into these parts: 1) training for optimisation during parameter C search 2) training for model construction 3) testing (representing future incoming data)

Using cross-validation and grid-search, I'd like to find the optimal C for my dataset. There seem to be different suggestions on how to partition data into these three parts

I could first divide my data into training (50%) and testing (50%). From the training, I use one half (25% of total data) for cross-validation for optimisation of C. Once I found the best C, I use THE OTHER HALF of the training data (25% of total data) , to construct the model using the best C. I then validate the model against the 50% testing data, the resulting accuracy is the final accuracy. This way, neither the data that went in for the model-construction nor the testing data have ever been used during optimisation.

Use 50% for training and 50% for testing. The entire 50% for training is used for cross-validation for optimisation of the parameter C. Once I found the best C, I train on the full 50% of the training data to obtain the model, and validate it on the test data.

I think the advantage of 1. is that neither model construction and testing data were in the optimisation process, hence there is less of an overfitting problem. I read differing accounts online though, so any input would be greatly appreciated. Does the method depend on how much data there is available? In my case, I don't have that many instances, so I am not sure if method 1. is recommended. Reference to other relevant sources/textbooks would also be greatly appreciated.

Answer 1

The standard approach for support vector machine tuning is to tune on the full training set through cross-validation, e.g. your second approach. We use cross-validation to ensure that training and testing is always done on independent data. There is no need to split the training set further just for tuning. The problem of overfitting is already alleviated through cross-validation.

You won't get any benefit from making a separate tuning set as per your first approach. In fact, your final model is likely to be worse, considering you would use less data to train the final model.

The main context in which tuning sets are used for SVMs is when the full training set is huge which would make the tuning phase last way too long.

Answer 2

With 120 cases in total, I'd try to avoid multiple splitting or at least don't make more splits than absolutely necessary. The problem behind this is that the precision of your error (or performance) estimation depends on the number of cases you can use for testing. The problem is that you need many test samples both for the optimization and for the validation of the final model, particularly if you go for classification-type error measures like hit rate, error rate, sensitivities, ... (which are proportions): they have a large variance.

E.g. if you go for a nested cross validation with 10 folds in the outer loop that means you do the optimization of C with an inner cross validation with a total of 108 cases. The random uncertainty on any C estimate you have is $\geq$ the variance due to the fact that only 108 cases are available.
To get a feeling what this means, play a bit around with confidence intervals for proportions and difference-of-proportions-tests (e.g. McNemar: your tests can be done in a paired fashion).

So here are my recommendations:

• go for a resampling scheme: out-of-bootstrap or iterated (aka repeated) $k$-fold cross validation.
• if you can, use external knowledge (e.g. experience from previous experiments / data sets / understanding of the classification problem) to actually fix C, it is probably a good idea to do that: optimization with classification-type error measures is almost impossible with just 120 samples (have a look for literature about cross validation for model optimization).
• if you cannot avoid grid search for C, use nested (aka double) resampling.
• in that case you should definitively have a look how much the optimal Cs differ between the different folds of the outer validation loop. Also have a look whether the different model qualities observed in the grid search are actually different enough so that you know at least that the chosen C is really better than the limits of C for the grid search.

Literature: Filzmoser, P. and Liebmann, B. and Varmuza, K.: Repeated double cross validation Journal of Chemometrics, 2009, 23, 160-171