Given a classifier (SVM) classifying in 2 'classes' (+1 or -1) for prediction purposes. It has an AUC score of 0.28, meaning its success rate is lower than just random predictions.

If I just do the opposite (ie: classifier says it'll be -1, so I'll assume it'll be +1 instead), does that mean my success rate in predicting will be about 72% (1-0.28)?

That doesn't seem very logical to me. Please explain to me how I should interpret this instead and why I can't just do the opposite of the classifier's predictions to get a higher success rate.

  • 1
    $\begingroup$ Because you have only 2 classes, if it fails to classify to this class, then it means it wins to classify to the other class. $\endgroup$ – ttnphns Jun 18 '14 at 5:23
  • $\begingroup$ So my original interpretation would be right? I was just sceptical because 70% successful predictions (if I were to do the opposite of what the classifier tells me to) is very high in the context I use it in. $\endgroup$ – Glenn Jun 18 '14 at 5:55

This interpretation is correct indeed. Here are a few candidate explanations why your classifier is apparently performing worse than random:

  1. Your classifier is actually random (true AUC close to 0.5). Your test set was small so 0.28 is within some confidence interval around 0.5. AUC can have pretty large 95% confidence intervals.
  2. Your classifier is over-fitted on the training set and performs very poorly on the test set.
  3. Your classifier is giving you the probability that the class is -1. Thus, you get a prediction (close to) 0 for a class 1, and 1 for a class -1 prediction. If your ROC method expects positive (+1) predictions to be higher than negative (-1) ones, you get a reversed curve.
  4. You have a bug somewhere. For instance it is not uncommon for classifiers to expect classes as 0 and 1 and I saw some implementations that can't deal with -1 or 2. Or you introduced an error somewhere, or something else along those lines.
  • $\begingroup$ As I use the same classifier on two other similar data sets (same size) and those have an AUC of around 0.5, is your option 4 most likely? The main difference between those data sets and this one is that the values here are in general much smaller. Could this offer an explanation why the AUC (and thus SVM (linear) classifications) are much worse for this data set? Though my test set of 20 data points would make your option 1 feasible too. (validation set to combat overfitting was used, so I don't think option 2 is it. Nor option 3 as it worked for other data sets). $\endgroup$ – Glenn Jun 18 '14 at 8:15
  • $\begingroup$ If anything, your comment would make option 1 the most likely. 20 is a very small sample size for ROC analysis, and 0.28 lies around limits of the 90% CI of a random ROC curve. Not surprising you get this kind of values once in a while. You should perform a power analysis of your study before drawing any conclusion about it. $\endgroup$ – Calimo Jun 18 '14 at 8:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.