0
$\begingroup$

I have a binary classification problem for which only $10$ positive samples are available for training. Negatives are in general in abundance, but I choose to use solely $70$ ($7$ negatives per one positive). I am trying to learn a kernel SVM (using the RBF kernel), thus I want to optimize the pair of parameters $C$, $\gamma$. I conduct grid search and I am wondering which division of the training set is more appropriate. Should I use $3$-, $5$-, or $10$-fold cross-validation? Something else maybe?

I am particularly interested in the case of $\mathbf{10}$-fold cross-validation, because I have only $10$ positive samples. Would that be a good approach?

$\endgroup$
  • 1
    $\begingroup$ With only 10 positives, I think it would be better to use repeated 5 fold cross-validation. Something like 10 times iterated 5 fold cv. $\endgroup$ – Marc Claesen Apr 18 '15 at 15:17
  • $\begingroup$ Thanks @MarcClaesen, the problem is that I cannot run so many iterations, because of complexity reasons (it's not a standard SVM), but would you think that a standard $5$-fold cross-validation procedure would be reasonable? $\endgroup$ – nullgeppetto Apr 18 '15 at 15:20
0
$\begingroup$

I think your sample size is so low, you can use sampling methods to increase you sample size (based on your case). Because of low sample size, I think it is better to use 5-fold cross validation but this depends on your case. The best solution for your answer is to evaluate your trained model (different cross validations and different obtained parameters) by out of sample data. This will give you a better insight for sampling and best cross validation techniques to use.

Besides this you can re-train your model and check range of obtained parameters. This will help you to find better range of parameters (by same cross validation or different folds for cross validation) and analyze effect of different cross validation techniques on you obtained results and parameters.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.