4
$\begingroup$

I am using R and e1071 package to tune a C-classification SVM.

My question is: regardless of the kernel type (linear, polynomial, radial basis or sigmoidal), is there any good criterion to choose the range in which cost and $\gamma$ parameters should range over and/or to choose what the granularity should be (that is, as an example, gamma = 10 ^ (1:2) or gamma = 1:2 or gamma = 100 ^ (1:2))?

I add a second question: can tune.svm() return the best kernel type, too?

Thanks,

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

Rather than using a grid search, it may be easier to use something like the Nelder-Mead simplex algorithm (which I believe has an R implementation) and use that to minimise the model selection criterion, and then you don't need to worry about the limits at all. Do minimise a continuous criterion though, for example the hinge loss on the test examples or radius-margin bound, rather than the test error rate (as that will be rather noisy and hard to optimised. Don't optimise the hyper-parameters too much though as it is also possible to get overfitting in model selection as well as in fitting the SVM.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thank you for your suggestment, Dikran Marsupial. Let you have to decide which is the best kernel type and the best value of cost and $\gamma$ parameters: what would you optimize first, according to which criterion and why? $\endgroup$
    – Lisa Ann
    Nov 21, 2012 at 11:06
  • 1
    $\begingroup$ Linear and RBF kernels are good reasonable choices to look at first, however the choice of kernel is very much data-dependent. The Nelder Mead method can optimise both parameters simultaneously, which is generally the best option. I rarely use SVMs these days, as Kernel Logistic Regression gives estimates of probability, which I find more useful than a binary decision for most real-world problems. Alternatively I use kernel ridge regression (AKA LS-SVM). The training procedure is much more straightforward, and you can get cross-validation almost for free, which I use for model selection. $\endgroup$
    – Dikran Marsupial
    Nov 21, 2012 at 12:27
  • $\begingroup$ Is the Kernel Logistic Regression like a non parametric Logit model (or similar to that one)? I will try it, thank you. Could you suggest me any R package with KLR? $\endgroup$
    – Lisa Ann
    Nov 21, 2012 at 12:33
  • 1
    $\begingroup$ KLR is essentially just a logistic regression model made non-linear using the same "kernel trick" used in the SVM. I am a MATLAB user, so I'm afraid I can't recommend a suitable R package. I've been meaning to learn R, so I may write one, one day! $\endgroup$
    – Dikran Marsupial
    Nov 21, 2012 at 13:21
  • 1
    $\begingroup$ @Dougal I suspect this is a "no free-lunch" issue, and there is unlikely to be any method that dominates a-priori. They key issue is to avoid optimising the hyper-parameters too much, which can be done by both frequentist and Bayesian approaches. $\endgroup$
    – Dikran Marsupial
    May 15, 2015 at 8:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.