9
$\begingroup$

When using libsvm, the parameter $\gamma$ is a parameter for the kernel function. Its default value is setup as $$\gamma = \frac{1}{\text{number of features.}}$$

Is there any theoretical guidance for setting up this parameter besides existing methods, e.g., grid search?

$\endgroup$

2 Answers 2

8
$\begingroup$

I would suggest the following theoretical guidance. When you are using Gaussian RBF kernel, your separating surface will be based on a combination of bell-shaped surfaces centered at each support vector. The width of each bell-shaped surface will be inversely proportional to $\gamma$. If this width is smaller than the minimum pair-wise distance for your data, you essentially have overfitting. If this width is larger than the maximum pair-wise distance for your data, all your points fall into one class and you don't have good performance either. So the optimal width should be somewhere between these two extremes.

$\endgroup$
1
  • $\begingroup$ pair-wise distance for your data = simple Euclidean distance after scaling? $\endgroup$
    – ihadanny
    Commented Jan 26, 2015 at 13:10
5
$\begingroup$

No, it is essentially data dependent. Grid search (over log-transformed hyper-parameters) is a very good method if you only have a small number of hyper-parameters to tune, but don't make the grid resolution too fine or you are likely to over-fit the tuning criterion. For problems with a larger number of kernel parameters, I find the Nelder-Mead simplex method works well.

$\endgroup$
2
  • $\begingroup$ Dikran, thanks for the answer. Can you elaborate more on the "data-dependent"? What's the relationship between r and a set of data? Or in other words, given a set of data, is there a way to define r based on these data? $\endgroup$
    – user3269
    Commented Sep 20, 2012 at 21:48
  • 1
    $\begingroup$ Essentially "data dependent" just means the best settings will vary according to the particular structure of the data and there isn't generally any better way to set them than minimising the cross-validation error. Kernel methods could really do with more theoretical analysis of how to go about learning the kernel, but unfortunately this is mathematically very difficult. $\endgroup$ Commented Sep 21, 2012 at 9:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.