2
$\begingroup$

I am training an SVM and I have 40k Negative Samples and 17k Positive samples. What I did is that I have divided my samples into training and testing subsets. In order to train the SVM I have used some of the training (not all) and I was randomly picking samples and apply the SVM into all testing data. I have found that if I use a specific amount of training data (not all) I get a very good performance over the testing data (0.99 sensitivity and 0.99 specificity).

How do I use k fold cross-validation now? I am confused. As I understood it, in k fold all the available data are used and they are divided in 5 subsets etc. Which will be the final SVM that I will use in 'real time'? The one of that I have found with my own good results?

I am using MATLAB (svmtrain, svmclassify, classperf).

$\endgroup$
2
$\begingroup$

I do not think that is the standard order of operations in cross-validation and you may have overfitting on the test set. Briefly, I believe that the standard practice is to:

  • Separate the testing set and put it to one side. Treat that testing data as the equivalent of an as-yet-unknown future which you will test your final model against.
  • Partition your training set into the $k$-folds, so if $k=5$ then split the training set into five parts.
  • Train your model (in effect decide hyperparameters, perhaps in your SVM the soft constraint parameter $C$ and the kernel parameter $\gamma$) by looking at how it performs with different hyperparameters when you run it on the combination four parts and validate against the fifth part - each run doing this five times, against each of the folds. Choose the hyperparameters which give the optimal results in this cross-validation.
  • Run your model, using the chosen hyperparmeters, on the whole training set. This is your final model.
  • Test (once only) your final model on the test set to see how accurate/sensitive/specific it is on what is designed to represent out-of-sample data.
$\endgroup$
  • $\begingroup$ (+1) The natural extension of this outline is more expensive in terms of computing power: do the cross-validation within the cross-validation, so that you rotate through testing sets and average performance over the test sets. But that means fitting many, many models. $\endgroup$ – Sycorax May 2 '15 at 21:41
  • $\begingroup$ ... which is known as double or nested cross validation. $\endgroup$ – cbeleites May 3 '15 at 8:28
  • $\begingroup$ Thanks for your quick answers. I did what Henry suggested.I did a 5 fold validation in my training set with perfect results but when i use the validation set i get very bad performance. What could be wrong?I am using a linear kernel (svmtrain(trainingFeatures, trainingLabels,'Kernel_Function','linear',... 'QuadProg_Opts',optimset('MaxIter',20000),'boxconstraint',1,'tolkkt',(1e-3),'kktviolationlevel', 0);)So i guess the only parameter to tune is 'C'and in matlab's svm case is the boxconstraint.Any Ideas what is wrong? $\endgroup$ – John May 4 '15 at 7:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.