5
$\begingroup$

I'm doing a binary classification using SVM classfier, libsvm, where roughly 95% belongs to one class.

The parameters C and gamma are to be set before the actual training takes place. I followed the tutorial but still can't get any good results.

There is a script that comes with the library that is supposed to help with choosing the right values for parameters but what this script is doing is basically maximizing the accuracy metric (TP+TN)/ALL, so in my case it chooses the parameters to label all data with prevailing class label.

I would like to choose parameters with recall and precision based metrics. How could I approach this problem. Accuracy is a meaningless metric for what I'm doing. Also I'm keen on changing the library libsvm to any other one that can help me with this problem as long as it takes data in the same format.

1 1:0.3 2:0.4 ... -1 1.0.4 2:0.23 and so on

Can anybody help?

UPDATE: yes I did try both grid.py and easy.py but even though grid search uses logarithmic scale it is extremely slow.I mean even if I run it on just small chunk of my data it takes tens of hours to finish. Is this the most efficient way to use SVM?? Have also tried svmlight but it does exactly the same it labels all data with one label.

UPDATE2: I reformed my question the better reflect what sort of issues I am facing

$\endgroup$
  • 1
    $\begingroup$ How many datapoints do you have? What kind of SVM are you uisng -- linear or Gaussian kernel (or some other kernel)? Kernel methods in libsvm are much slower. $\endgroup$ – SheldonCooper Mar 5 '11 at 20:08
  • $\begingroup$ C-SVC, 26k 9-dimensional vectors, polynomial kernel. Just edited grid.py to maximize F-scores instead, but still looking for some clever options to run SVM Classfier. $\endgroup$ – matcheek Mar 5 '11 at 20:24
5
$\begingroup$

I would do two things. First, to address your issue with accuracy due to imbalanced data, you need to set the cost of misclassifying positive and negative examples separately. A reasonable rule of thumb in your case would be to set the cost to 5 for the larger class and to 95 for the smaller class. This way misclassifying 10% of the smaller class will have the same cost as misclassifying 10% of the larger class even though the latter 10% is a much larger number of points. If you use the command line, the command is something like -w0 5 -w1 95. I feel this needs to be done anyway (even though you are using F score for now) because this is what SVM optimizes, so unless you do it, all your F scores will be poor.

Second, to address the issue of speed, I would try pre-computing the kernel. For 26k points this is borderline infeasible, but if you are willing to subsample, you can precompute the kernel once for each gamma and reuse it across all C's.

$\endgroup$
  • $\begingroup$ after applying weighted penalty for each class it works faster. Going through all the mathematical intricacies of SVM now. Thanks for help $\endgroup$ – matcheek Mar 8 '11 at 1:12
1
$\begingroup$

By Default libSVM find the optimal hyper-parameters, for the SVM model using cross validation methods and by using Accuracy (for classification), or Mean Square Error (for regression) as a measure for evaluation.

Weka has several other evaluation metric to find the optimal parameters (using the gridSearch)

If the metric you are interested is not there, the simplest solution that comes to mind is to write a little program that would perform cross validation, and optimize the parameters of the model based on your measure of choice.

$\endgroup$
  • $\begingroup$ I have made an approach to maximize F-Score using grid search but it's very expensive, almost infeasible. $\endgroup$ – matcheek Mar 5 '11 at 15:25
  • 1
    $\begingroup$ @matcheek This is the SVM reality -- you can try randomly reducing the dataset and optimizing on it, to at least get an approx. where to look. Or just change the classifier ;-) $\endgroup$ – user88 Mar 5 '11 at 16:03
  • 1
    $\begingroup$ The approach the WEKA uses to optimize the hyperparamters, is to: first: use a 2-fold cross validation to "roughly" estimate the location of the hyperparamters on the grid, and second: do a 10-folds cross validation on the adjacent points (found in the previous steps) to find a more "accurate" estimate of the hyperparameters.. Hope this help speed your code up.. :) $\endgroup$ – Lalas Mar 5 '11 at 17:34
  • $\begingroup$ Specialized libraries exist to perform tuning in a disciplined way, such as Optunity or HyperOpt. $\endgroup$ – Marc Claesen Nov 21 '14 at 8:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.