Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 3 down vote favorite
2

For the $\nu$-SVM (for both classification and regression cases) the $\nu \in (0;1)$ should be selected. The LIBSVM guide suggests to use grid search for identifying the optimal value of the $C$ parameter for $C$-SVM, it also recommends to try following values $C = 2^{-5}, 2^{-3}, \dots, 2^{15}$.

So the question is, are there any recommendations for values of the $\nu$ parameter in case of $\nu$-SVMs?

share|improve this question

2 Answers

up vote 3 down vote accepted

Rather than use a grid search, you can just optimise the hyper-parameters using standard numeric optimisation techniques (e.g. gradient descent). If you don't have estimates of the gradients, you can use the Nelder-Mead simplex method, which doesn't require gradient information and is vastly more efficient than grid-search methods. I would use the logit function to map the (0;1) range of $\nu$ onto $(-\infty;+\infty)$ to get an unconstrained optimisation problem.

If you really want to use grid search, then just spacing the evaluation points linearly in the range 0 - 1 should be fine.

share|improve this answer
Let me clarify your answer. So you suggest to use gradient descent like methods for SVM loss function minimization (that depends on $\nu, \alpha_1, \dots, \alpha_n$), instead of SMO method that optimizes loss function for fixed $\nu$. Right? To use gradient descent, the loss function should be unimodal. So does the loss function unimodal with respect to $\nu$ variable? – TheBug Apr 7 '11 at 8:29
@TheBug, no, the SVM itself is still trained using SMO, but the hyper-parameters ($\nu$ and any kernel parameters) are optimised by gradient descent (or Nelder Mead simplex) minimisation of some suitable model selection criterion (e.g. cross-validation error). At each step you have to train several SVMs with fixed $\nu$ in order to define the simplex or to compute the gradient (using finite differences if it can't be done analytically). The cross-validation error is not generally unimodal, but it is normally smooth and such methods work well in practice. – Dikran Marsupial Apr 7 '11 at 9:17
Thank you, now it's clear. – TheBug Apr 7 '11 at 9:58
One more question about this technique. Can be same method applied to find $C$ and $\nu$ for $\nu$-SV regression? – TheBug Apr 23 '11 at 12:56
It should be O.K., I haven't used $\nu$-SVR, but I regularly use these sorts of methods for optimising the hyper-parameters of a wide range of kernel models and Gaussian processes (both regression and classification). As long as you apply a suitable transformation to get an unconstrained optimisation problem, it seems pretty reliable. – Dikran Marsupial Apr 23 '11 at 13:00
up vote 0 down vote

well, actually there is an heuristic to get some meaningful start values, see the paper "A geometric interpretation of v-SVM" and references therein.

The idea is the following: "v is an upper bound on the fraction of margin errors, a lower bound on the fraction of support vectors, and both quantities approach v asymptotically". This number cannot exceed the quantity 2*lmin/l, where l is the total number of SVs and lmin is the minimum between the number of positive and negative SVs (labels +/-1).

Notice that this means that for inbalanced problems, you will have to work with lower values of v, so you will tend to overfit data.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.