12
$\begingroup$

I have 12 positive training sets (cancer cells treated with drugs with each of 12 different mechanisms of action). For each of these positive training sets, I would like to train a support-vector machine to distinguish it from a negative set of equal size sampled from the experiment. Each set has between 1000 and 6000 cells, and there are 476 features (image features) of each cell, each scaled linearly to [0, 1].

I use LIBSVM and the Gaussian RGB kernel. Using five-fold crossvalidation, I have done a grid search for log₂ C ∈ [-5, 15] and log₂ ɣ ∈ [-15, 3]. The results are as follows:

Results of grid search

I was disappointed that there is not a single set of parameters that give high accuracies for all 12 classification problems. I was also surprised that the grids do not generally show a high-accuracy region surrounded by lower accuracies. Does this just mean that I need to expand the search parameter space, or is the grid search an indication that something else is wrong?

Improve this question
$\endgroup$
  • 2
    $\begingroup$ Re disappointment: You wouldn't expect each problem to have the same parameters, so why would you expect the problems to share good values for the hyperparameters (log gamma and C)? $\endgroup$ – conjugateprior Jan 4 '11 at 19:19
  • $\begingroup$ @Conjugate Prior: The training sets are subsets of the same experiment, and the negative training sets are sampled from the same population, so I had hoped that the same RBF kernel width ɣ would be effective. Because the positive sets are being discriminated from the same background (negative) population, I had hoped that the ideal penalty C would be similar as well. If this is not the case, it makes SVM really hard to apply. Gentle boosting, for instance, seems much easier to tune. $\endgroup$ – Vebjorn Ljosa Jan 4 '11 at 19:51
  • $\begingroup$ Aha. But it seems to me that although it is the same experiment in the physical sense, you are nevertheless attacking separate and different problems in the statistical sense. Particularly if the negative cases are resampled for each treatment. $\endgroup$ – conjugateprior Jan 4 '11 at 19:59
  • 1
    $\begingroup$ BTW, grid search is rather inefficient, the Nelder-Mead simplex optimisation algorithm is very effective, as are gradient descent optimzation methods. Grid search is simple, but a bit "brute force". $\endgroup$ – Dikran Marsupial Jan 5 '11 at 0:59
  • $\begingroup$ @Vebjorn Ljosa (a year later), how much do the 5 values scatter, say at the final (C, gamma) ? Are the 12 plots all scaled the same, e.g. 50 % .. 100 % correct prediction ? Thanks $\endgroup$ – denis Feb 11 '12 at 14:33
9
$\begingroup$

The optimal values for the hyper-parameters will be different for different learning taks, you need to tune them separately for every problem.

The reason you don't get a single optimum is becuase both the kernel parameter and the regularisation parameter control the complexity of the model. If C is small you get a smooth model, likewise if the kernel with is broad, you will get a smooth model (as the basis functions are not very local). This means that different combinations of C and the kernel width lead to similarly complex models, with similar performance (which is why you get the diagonal feature in many of the plots you have).

The optimum also depends on the particular sampling of the training set. It is possible to over-fit the cross-validation error, so choosing the hyper-parameters by cross-validation can actually make performance worse if you are unlucky. See Cawley and Talbot for some discussion of this.

The fact that there is a broad plateau of values for the hyper-parameters where you get similarly good values is actually a good feature of support vector machines as it suggests that they are not overly vulnerable to over-fitting in model selection. If you had a sharp peak at the optimal values, that would be a bad thing as the peak would be difficult to find using a finite dataset which would provide an unreliable indication of where that peak actually resides.

Improve this answer
$\endgroup$
  • $\begingroup$ BTW I am performing a study on over-fitting in model selection using grid-search, which turns out to be far more interesting that I had thought. Even with few hyper-parameters, you can still over-fit the model selection criterion if you optimise over a grid that it too fine! $\endgroup$ – Dikran Marsupial Apr 20 '12 at 17:54
  • $\begingroup$ I am getting towards the end of the simulation work now, hopefully I'll be able to submit the paper in a month or two... $\endgroup$ – Dikran Marsupial Sep 5 '12 at 17:43
  • $\begingroup$ I'd be interested to read that paper if it is finished? I have come across some strange spikes etc in grid search optimisations which seems similar to what you discuss here. $\endgroup$ – BGreene Sep 21 '12 at 14:19
  • $\begingroup$ All the simulation work is now complete, I am just putting the paper together at the moment (mostly just ensuring that it is all fully reproducible). I have saved all of the grids so some reanalysis should be possible to look into other questions I didn't think of at the time. $\endgroup$ – Dikran Marsupial Sep 21 '12 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.