I'm having some trouble getting the NM Simplex to find a good minimum for selecting hyperparameters of a rbf SVC. Not only am I tuning the 2 SVC parameters (C and gamma) I also have five class weights that I'm trying to tune. Grid searches seem fairly infeasible in this case. The NM Simplex doesn't seem test inputs far away enough from the initial guess. I am transforming the inputs (e.g. the NM Simplex is searching in range i while 2^i is inputted as the C parameter for the SVC). Even so, the NM Simplex always seems to end up back at the initial guess.

Any suggestions or any other minimization algorithms that might be worthwhile?



Both C amd Gamma should be searched on logarithmic axes, and it is important to start from reasonable initial values (which will depend on the exact representation of the kernel etc). It could also be that you need to make the stopping criterion smaller. I regularly use NM simplex for tuning kernel machines, so I know from experience that it can be made to work. It is also possible to tune SVMs using gradient descent (see the work of Olivier Chapelle), if you don't want the additional programming, you can always approximate the gradients via finite differences (fminunc in MATLAB can do this for you).

  • $\begingroup$ Thanks for the info. I am searching C and gamma on a log base 2 scale. And the weights just straight up. The NM Simplex seems to be having some issues as it appears to go from small to fairly large values of C, causing the SVM to predict all one class vs. all another class. Not sure why this is the case. $\endgroup$ – tomas Mar 16 '12 at 20:43
  • $\begingroup$ The weights probably ought to be strictly positive as well, I'd use a log2 scale for those as well. If those are going negative I can see how that might cause things to go haywire. There is a parameter that controls the size of the initial simplex (Delta in my implementation - theoval.cmp.uea.ac.uk/matlab/optim/simplex.m ), maybe you need to make that a bit smaller (and maybe gamma, rho and sigma). $\endgroup$ – Dikran Marsupial Mar 17 '12 at 12:34
  • $\begingroup$ Thanks Dikran. Appreciate all the help. I'm still having some trouble (I increased the starting size of the simplex so the vertices would have diff function values, but now it's causing the C parameter to go off into infinity) but I was wondering about another thing, which was I know there's a way to a priori determine the class weights from the training data. Do you know what method this is or what paper it's from? Thanks. I realize that X validation is probably preferable but wanted to take a look nonetheless. $\endgroup$ – tomas Mar 20 '12 at 18:45
  • $\begingroup$ I was one of many to have written a paper on this long ago, you can find mine here: theoval.cmp.uea.ac.uk/publications/pdf/ijcnn2001.pdf However, one of the problems with these adjustments is that they are only assymptotically correct in the limit of an infinite training sample. In practice the same is usually too small for this to work as well as I'd like, so I have found that it is better to have a different C for each class and optimise them via cross-validation. $\endgroup$ – Dikran Marsupial Mar 21 '12 at 12:22
  • $\begingroup$ Thanks Dikran. I will check out your paper. I definitely think cross validation is the preferable method, it's just been difficult trying to balance between my ideal modeling approach and computational complexity, which I suppose is often the case in machine learning. $\endgroup$ – tomas Mar 21 '12 at 20:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.