0
$\begingroup$

I have a very unbalanced binary classification problem (positive class: 0.2%). I need to evaluate it using f1 of the positive class. Now, I'm doing some baselines using an SVM. What I get is a relatively low f1 (0.61) and a very high AUC (0.98). If I understood it correctly, this means that the classifier correctly ranks my examples, but it uses an operating point that is not optimal. So, my questions are:

  1. Is my understanding correct?
  2. Is SVM using a threshold of 0.5?
  3. Does it make sense (both mathematically and from a ML point of view) to tune this parameter, for example by using cross-validation?
$\endgroup$
  • 1
    $\begingroup$ May I ask you what package you are using? It is not necessarily straight forward to draw the AUC curve for an SVM, as you don't have a probability threashold - only thing you can do is change the bias term $\endgroup$ – Davide ND Feb 4 at 15:52
  • 1
    $\begingroup$ See this page for related discussion about metrics for classification models. As @DavideND wrote, a proper AUC requires a probability model, not the all-or-none choices made by an SVM in its simplest implementation. Some implementations of SVM can provide probability estimates, as noted on the linked page. You, rather than the program, should be making the choice of probability threshold, based on the relative costs of false-positive and false-negative findings. $\endgroup$ – EdM Feb 4 at 16:20
  • $\begingroup$ @EdM thanks for the comment - some really interesting links in there! $\endgroup$ – Davide ND Feb 5 at 10:22
1
$\begingroup$

Is my understanding correct?

Yes, in general I would say that it is, and that your classifier might be using some different parameters to work better with your unbalanced classes.
However, AUC curve with an SVM might not be completely "truthful", and this brings me to the next point

Is SVM using a threshold of 0.5?

First of all, I suggest reading this post and its answer.

SVMs do not make a probability model, but optimize the position of the separating hyperplane between the classes. Therefore classification does not use a probability threshold, since they do not model probability at all.

The AUC curve can be obtained either using some "synthetic "predict_proba method or using the decision_function one.
The first one is not necessarily stable, and documentation suggests that it is not even guaranteed to match the results of the classification itself (meaning that there is no clear threshold). The second one is based on distances from the hyperplane, and you could find ROC points by "varying" the bias term and moving the hyperplane. However this is a parameter that is optimized during the fitting, so in my opinion tuning it manually afterwards seems a bit weird (however, I am not an SVM expert so someone might have a say on this).

Does it make sense (both mathematically and from a ML point of view) to tune this parameter, for example by using cross-validation?

Linking to the previous point, I would not use the probabilities (also, outputting them makes the fitting much longer) nor I would manually tweak the bias term. I think one way to try and use a different operating point would be to tune the class_weight and sample_weight parameters, which should allow you to deal with unbalanced classes.

Edit:
Thanks @EdM for the very interesting comment. Indeed, in one of the papers suggested in that question (this one) the author changes the bias parameter b manually from the one that better separates the classes to the one that obtains the best balance in recall and precision. However, as it mentions, the classifier is not technically an SVM anymore, as it optimizes a different function.

$\endgroup$
  • $\begingroup$ Thanks a lot! Can you check the link to the paper? Looks like it's broken, and it seems interesting. $\endgroup$ – ImAUser Feb 5 at 11:27
  • $\begingroup$ Link fixed! (please consider marking the question as solved if you find the answer satisfactory, so it doenst appear as unanswered) $\endgroup$ – Davide ND Feb 5 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.