Cross-Validation in binary classification using only 10 positive samples (SVM) I have a binary classification problem for which only $10$ positive samples are available for training. Negatives are in general in abundance, but I choose to use solely $70$ ($7$ negatives per one positive). I am trying to learn a kernel SVM (using the RBF kernel), thus I want to optimize the pair of parameters $C$, $\gamma$. I conduct grid search and I am wondering which division of the training set is more appropriate. Should I use $3$-, $5$-, or $10$-fold cross-validation? Something else maybe?
I am particularly interested in the case of $\mathbf{10}$-fold cross-validation, because I have only $10$ positive samples. Would that be a good approach?
 A: Rather than using cross-validation to tune the hyper-parameters, optimise a bound on the generalisation error, such as the Span bound (or the radius-margin bound).  This can also be optimised using gradient descent, or the Nedler-Mead simplex method, which are likely to be rather faster than grid-search. See:
Chapelle, O., Vapnik, V., Bousquet, O. et al. Choosing Multiple Parameters for Support Vector Machines. Machine Learning 46, 131–159 (2002). doi:10.1023/A:1012450327387
For such a small dataset, I would use a Least-Squares Support Vector Machine, for which the leave-one-out error can be calculated at negligible computational cost, as a by-product of the training algorithm.  I would optimise the leave-one-out estimate of the mean-squared error (also known as the Brier score, or Allen's PRESS statistic).  See my paper:
G. C. Cawley, "Leave-One-Out Cross-Validation Based Model Selection Criteria for Weighted LS-SVMs," The 2006 IEEE International Joint Conference on Neural Network Proceedings, 2006, pp. 1661-1668, doi:10.1109/IJCNN.2006.246634 (pre-print).
If computational expense is not an issue, I would use bootstrapping rather than cross-validation as a model selection criterion, as it is likely to have a lower variance than cross-validation.
A: A sample size of 96 is required just to estimate a single proportion with a decent (+- 0.1) margin of error.  10 positives is insufficient for multivariable analysis.  Present descriptive results and stop.  No amount of resampling (e.g. cross-validation) can help.  The sample size is also inadequate for (1) finding the optimum penalty and (2) estimating predictive accuracy.
