# Training an SVM and performing cross validation

I am training an SVM and I have 40k Negative Samples and 17k Positive samples. What I did is that I have divided my samples into training and testing subsets. In order to train the SVM I have used some of the training (not all) and I was randomly picking samples and apply the SVM into all testing data. I have found that if I use a specific amount of training data (not all) I get a very good performance over the testing data (0.99 sensitivity and 0.99 specificity).

How do I use k fold cross-validation now? I am confused. As I understood it, in k fold all the available data are used and they are divided in 5 subsets etc. Which will be the final SVM that I will use in 'real time'? The one of that I have found with my own good results?

I am using MATLAB (svmtrain, svmclassify, classperf).

I do not think that is the standard order of operations in cross-validation and you may have overfitting on the test set. Briefly, I believe that the standard practice is to:

• Separate the testing set and put it to one side. Treat that testing data as the equivalent of an as-yet-unknown future which you will test your final model against.
• Partition your training set into the $k$-folds, so if $k=5$ then split the training set into five parts.
• Train your model (in effect decide hyperparameters, perhaps in your SVM the soft constraint parameter $C$ and the kernel parameter $\gamma$) by looking at how it performs with different hyperparameters when you run it on the combination four parts and validate against the fifth part - each run doing this five times, against each of the folds. Choose the hyperparameters which give the optimal results in this cross-validation.
• Run your model, using the chosen hyperparmeters, on the whole training set. This is your final model.
• Test (once only) your final model on the test set to see how accurate/sensitive/specific it is on what is designed to represent out-of-sample data.
• (+1) The natural extension of this outline is more expensive in terms of computing power: do the cross-validation within the cross-validation, so that you rotate through testing sets and average performance over the test sets. But that means fitting many, many models.
– Sycorax
May 2, 2015 at 21:41
• ... which is known as double or nested cross validation. May 3, 2015 at 8:28
• Thanks for your quick answers. I did what Henry suggested.I did a 5 fold validation in my training set with perfect results but when i use the validation set i get very bad performance. What could be wrong?I am using a linear kernel (svmtrain(trainingFeatures, trainingLabels,'Kernel_Function','linear',... 'QuadProg_Opts',optimset('MaxIter',20000),'boxconstraint',1,'tolkkt',(1e-3),'kktviolationlevel', 0);)So i guess the only parameter to tune is 'C'and in matlab's svm case is the boxconstraint.Any Ideas what is wrong?
– John
May 4, 2015 at 7:37