3
$\begingroup$

For the $\nu$-SVM (for both classification and regression cases) the $\nu \in (0;1)$ should be selected. The LIBSVM guide suggests to use grid search for identifying the optimal value of the $C$ parameter for $C$-SVM, it also recommends to try following values $C = 2^{-5}, 2^{-3}, \dots, 2^{15}$.

So the question is, are there any recommendations for values of the $\nu$ parameter in case of $\nu$-SVMs?

$\endgroup$

3 Answers 3

5
$\begingroup$

Rather than use a grid search, you can just optimise the hyper-parameters using standard numeric optimisation techniques (e.g. gradient descent). If you don't have estimates of the gradients, you can use the Nelder-Mead simplex method, which doesn't require gradient information and is vastly more efficient than grid-search methods. I would use the logit function to map the (0;1) range of $\nu$ onto $(-\infty;+\infty)$ to get an unconstrained optimisation problem.

If you really want to use grid search, then just spacing the evaluation points linearly in the range 0 - 1 should be fine.

$\endgroup$
4
  • $\begingroup$ Let me clarify your answer. So you suggest to use gradient descent like methods for SVM loss function minimization (that depends on $\nu, \alpha_1, \dots, \alpha_n$), instead of SMO method that optimizes loss function for fixed $\nu$. Right? To use gradient descent, the loss function should be unimodal. So does the loss function unimodal with respect to $\nu$ variable? $\endgroup$
    – TheBug
    Apr 7, 2011 at 8:29
  • $\begingroup$ @TheBug, no, the SVM itself is still trained using SMO, but the hyper-parameters ($\nu$ and any kernel parameters) are optimised by gradient descent (or Nelder Mead simplex) minimisation of some suitable model selection criterion (e.g. cross-validation error). At each step you have to train several SVMs with fixed $\nu$ in order to define the simplex or to compute the gradient (using finite differences if it can't be done analytically). The cross-validation error is not generally unimodal, but it is normally smooth and such methods work well in practice. $\endgroup$ Apr 7, 2011 at 9:17
  • $\begingroup$ One more question about this technique. Can be same method applied to find $C$ and $\nu$ for $\nu$-SV regression? $\endgroup$
    – TheBug
    Apr 23, 2011 at 12:56
  • $\begingroup$ It should be O.K., I haven't used $\nu$-SVR, but I regularly use these sorts of methods for optimising the hyper-parameters of a wide range of kernel models and Gaussian processes (both regression and classification). As long as you apply a suitable transformation to get an unconstrained optimisation problem, it seems pretty reliable. $\endgroup$ Apr 23, 2011 at 13:00
3
$\begingroup$

well, actually there is an heuristic to get some meaningful start values, see the paper "A geometric interpretation of v-SVM" and references therein.

The idea is the following: "v is an upper bound on the fraction of margin errors, a lower bound on the fraction of support vectors, and both quantities approach v asymptotically". This number cannot exceed the quantity 2*lmin/l, where l is the total number of SVs and lmin is the minimum between the number of positive and negative SVs (labels +/-1).

Notice that this means that for inbalanced problems, you will have to work with lower values of v, so you will tend to overfit data.

$\endgroup$
2
$\begingroup$

Note that $\nu$-SVM and $C$-SVM are equivalent (at least for balanced problems). Performing a grid-search over $C$ should be more than sufficient.

Also, generally $C$-SVM is more efficient to solve (note e.g. for $\nu$-SVM in libSVM that the initial starting point will tend to set many non-support vectors to have bound $\alpha_i$, thus loading large parts of the kernel matrix that you may not need).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.