1
$\begingroup$

I have a binary classification problem for which only $10$ positive samples are available for training. Negatives are in general in abundance, but I choose to use solely $70$ ($7$ negatives per one positive). I am trying to learn a kernel SVM (using the RBF kernel), thus I want to optimize the pair of parameters $C$, $\gamma$. I conduct grid search and I am wondering which division of the training set is more appropriate. Should I use $3$-, $5$-, or $10$-fold cross-validation? Something else maybe?

I am particularly interested in the case of $\mathbf{10}$-fold cross-validation, because I have only $10$ positive samples. Would that be a good approach?

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ With only 10 positives, I think it would be better to use repeated 5 fold cross-validation. Something like 10 times iterated 5 fold cv. $\endgroup$
    – Marc Claesen
    Commented Apr 18, 2015 at 15:17
  • $\begingroup$ Thanks @MarcClaesen, the problem is that I cannot run so many iterations, because of complexity reasons (it's not a standard SVM), but would you think that a standard $5$-fold cross-validation procedure would be reasonable? $\endgroup$
    – nullgeppetto
    Commented Apr 18, 2015 at 15:20

2 Answers 2

Reset to default
3
$\begingroup$

Rather than using cross-validation to tune the hyper-parameters, optimise a bound on the generalisation error, such as the Span bound (or the radius-margin bound). This can also be optimised using gradient descent, or the Nedler-Mead simplex method, which are likely to be rather faster than grid-search. See:

Chapelle, O., Vapnik, V., Bousquet, O. et al. Choosing Multiple Parameters for Support Vector Machines. Machine Learning 46, 131–159 (2002). doi:10.1023/A:1012450327387

For such a small dataset, I would use a Least-Squares Support Vector Machine, for which the leave-one-out error can be calculated at negligible computational cost, as a by-product of the training algorithm. I would optimise the leave-one-out estimate of the mean-squared error (also known as the Brier score, or Allen's PRESS statistic). See my paper:

G. C. Cawley, "Leave-One-Out Cross-Validation Based Model Selection Criteria for Weighted LS-SVMs," The 2006 IEEE International Joint Conference on Neural Network Proceedings, 2006, pp. 1661-1668, doi:10.1109/IJCNN.2006.246634 (pre-print).

If computational expense is not an issue, I would use bootstrapping rather than cross-validation as a model selection criterion, as it is likely to have a lower variance than cross-validation.

Improve this answer
$\endgroup$
3
$\begingroup$

A sample size of 96 is required just to estimate a single proportion with a decent (+- 0.1) margin of error. 10 positives is insufficient for multivariable analysis. Present descriptive results and stop. No amount of resampling (e.g. cross-validation) can help. The sample size is also inadequate for (1) finding the optimum penalty and (2) estimating predictive accuracy.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Where did they mention anything about estimating a proportion? $\endgroup$
    – astel
    Commented Sep 12, 2021 at 12:15
  • $\begingroup$ @astel Accuracy and many similar metrics are proportions. $\endgroup$
    – Dave
    Commented Sep 12, 2021 at 16:03
  • $\begingroup$ @Dave, I don't think accuracy was mentioned in the question. Personally, I don't agree that it is ever the case that you don't have enough data for an analysis. The level of uncertainty where the analysis ceases to be useful depends on the purpose of the analysis. From that perspective, a Gaussian process classifier might be a better choice than the SVM for such a small problem as it is easier to characterise the uncertainty (and probably seek to integrate over the hyper-parameters rather than optimise them). $\endgroup$
    – Dikran Marsupial
    Commented Sep 14, 2021 at 7:17
  • $\begingroup$ You'll be surprised at the instability of the result, and of the wide uncertainty estimates in predictions, when computed unbiasedly. $\endgroup$
    – Frank Harrell
    Commented Sep 14, 2021 at 11:51
  • $\begingroup$ @Dave, as Dikran mentions the question doesn't mention accuracy at all, but even if it did the formula Frank uses to arrive at 96 samples assumes a true proportion of 0.5 and unless you expect your model to be completely random, I think you can safely assume the true proportion to be greater than that. Also, the OP mentions having an abundance of negative samples with which to use so even if you used Franks requirement of 96 samples, the OP has that. My main issue with his answer is it reads like if you don't have 96 samples just stop what you are doing! With no justification whatsoever $\endgroup$
    – astel
    Commented Sep 14, 2021 at 19:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.