4
$\begingroup$

I am using grid search in order to find the best values for the SVM parameters (namely C and gamma). The search is performed in two steps: coarse grid search and finer grid search, as prescribed in this "A Practical Guide to Support Vector Classification" article.

What is not clear to me is figure 2 and figure 3 in the aforementioned article: What are the name/type of these figures and how should they prepared? One thing that I am not entirely sure about is, as shown in figure 2, there are many possible pairs of (C, gamma) values that produce the same cross-validation rate (i.e. 77.5). So, based on what the best (C, gamma) in this coarse search step was determined? Is the 77.5, as I think, just a rounded number?

Also, are there other (better?) figure types that can be used to visualize the grid search steps?

$\endgroup$
1
  • $\begingroup$ This looks like level curves of a function of 2 variables. There exist algorithms to generate these based on values on a grid. $\endgroup$
    – Gumeo
    Commented Sep 14, 2015 at 14:02

2 Answers 2

2
$\begingroup$

The figures you refer to are contour plots of the hyperparameter response surface, that is the cross-validated performance for each pair of hyperparameters. Visualizing grid search is effectively only useful when you have a pair of hyperparameters, which is rarely the case.

Grid search is generally a poor approach to optimize hyperparameters, though it is still by far the most common approach. Better approaches use metaheuristic or Bayesian solvers and are available in packages like Optunity and Hyperopt .

Here are some examples of SVM hyperparameter optimization using Optunity:

Disclaimer: I'm the main developer of Optunity.

Edit: only realize now that you asked about alternative figure types instead of alternative optimization approaches, so I guess part of this answer is not directly relevant. That said, I think the suboptimality of grid search can't be stressed enough.

$\endgroup$
2
$\begingroup$

Fig. 2 and 3 are hyperparameter response surfaces. The diagrams are a level plot for a function of two variables. The authors computed values of the SVM performance at a grid of hyperparamter tuples, and then some plotting algorithm identified level sets.

A common choice of the best hyperparameter tuple is to choose the more strongly regularized model within 1 standard error of the optimum. This is called the 1 standard error rule, and it is described in Elements of Statistical Learning. This is done to avoid choosing an overly optimistic model.

Another way to display the hyperparameter response surface is a false color diagram, where the color of a particular polygon (usually a rectangle) corresponds to the value of the performance metric at the hyperparameter tuple. But it's only possible to construct these diagrams for very small numbers of hyperparameters, as constructing plots in a larger number of dimensions is impossible.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.