3
$\begingroup$

I use two libraries to perform SVM: liblinear and sofia (implements PEGASOS). I train them both on the same data (linear SVM), and then test them on two different data sets. The accuracy is very different:

In one test set, liblinear accuracy is 95% whereas sofia accuracy is 80% (this test test is made out of only positive labels) In the other data set, liblinear accuracy is 90% whereas sofia accuracy is 97% (this test test is made out of only negative labels)

The both perform similarly over the balanced train data, with 95% accuracy in 10-fold CV.

Does that make any sense?

$\endgroup$
  • $\begingroup$ Have u made sure you shuffled your training and testing datasets for using both packages to do the testing? Also, you should use the same set of hyperparameters (e.g. C) in doing the comparison. $\endgroup$ – Frank Puk Dec 9 '15 at 7:00
2
$\begingroup$

Evaluating your models based on a test set with only one class is a bad idea. What do you expect to learn from this?

Secondly, discrete metrics like accuracy, precision, recall are poor choices to assess which model is best. It is far better to use metrics like area under the ROC/PR curve.

Assuming you are talking about the same data set, then based on your description it is quite clear that the LIBLINEAR model is less conservative, e.g. has a lower threshold for positive predictions (that is, it predicts more positives). Hence, the LIBLINEAR model has higher TPR (= higher 'accuracy' on your first test set with only positives) and higher FPR (= lower 'accuracy' on your second test set with only negatives).

For all you know, both models might be identical except for a translation (this is quite common for linear SVM), so in terms of AUC they would be identical, while discrete measures like accuracy may mislead you into believing that the models themselves are substantially different.

Additionally, did you use the same hyperparameters when making these comparisons (that is, the same regularization and misclassification penalties per class in both toolboxes)? If your models are parameterized differently, then ofcourse their performance is different. That has nothing to do with the software, though.

$\endgroup$
  • $\begingroup$ Thanks for the answer. I use the parameters that resulted in the best 10 fold CV error on the train set which is a balanced one. I use the default threshold for the prediction label, which is 0.5. I will look at the ROC curve. $\endgroup$ – Roy Jun 17 '15 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.