0
$\begingroup$

I am using support vector machines for an unbalanced binary problem (0: 25%, 1: 75%). I do K-fold cross-validation with $K=10$. The metrics I get are:

  • 80% classification accuracy on average for the training set
  • 70% classification accuracy on average for the test set
  • 50% f1-score on average for the test set.

What kind of interpretation I can do here? In particular, is the f1-score a good one or a not so good one ("publishable")?

I am mainly interested in correctly classifying the instances of class '0'. What metrics should I be looking at?

$\endgroup$
1
$\begingroup$

Firstly, I hope you used stratified cross-validation for your unbalanced dataset (if not, you should seriously consider it, see my response here). Second, there is no absolute performance metric. Each metric is going to tell you something slightly different. You have more questions to ask yourself. If we consider ‘0’ as the class we want to correctly classify do you care more about not misclassifying ‘0’ as a ‘1’ or are you also concerned about misclassifying ‘1’ as a ‘0’?

Accuracy is nice because it is so approachable. However, when it comes to unbalanced datasets you can run in to problems. To take your 25/75 example (with 100 samples for simplicity), let’s say the model classified only three ‘0’s correctly. You would have the following confusion matrix.

    0   1
0   3   22
1   0   75

If I calculate accuracy (3+75)/100, it would come to 78% Accuracy. THIS IS HUGE FOR MANY MODELS!!! But clearly if you are interested in classifying ‘0’s this is a terrible model. Taking in to account the false negatives with the F1-measure $2TP/(2TP + FP + FN)$ comes to ~0.214. Clearly this is telling us that this model isn’t performing very well.

Since you say you are primarily concerned with classifying ‘0’s, you would also want to report the Positive Predictive Value ($PPV$) where $PPV = TP/(TP+FP)$, which comes to 0.12 which tells you that you are really doing bad with respect to classifying ‘0’s

Other ‘comprehensive’ metrics include the Cohen’s Kappa and Area Under the Curve (AUC) if you wish to explore others.

General summary points

  1. Clarify what you goal is (broad classifier or just ‘0’s).
  2. Look at your confusion matrix (gives you a more natural feel of your model’s performance)
  3. Choose your model performance metric(s) and report what the metric represents. In your case, the F1-measure is not looking very favorable for you model.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.