0
$\begingroup$

I have two sets (small) of features that I need to compare against a class, and decide which set is better at classifying. The features are generated using ranking methods, and only the top 2% of the features are compared, to determine which method is better differentiating amongst the top ranking features. Both the small datasets achieve a low AUC: 0.2 and 0.15 to be exact?

Could I make a conclusion that Method 1 is better than Method 2 in differentiating amongst the top 2% basis on the scores if both AUC values are below 0.5?

$\endgroup$
  • 2
    $\begingroup$ Unless you have a different definition of AUC I think this means that they get it wrong better than chance. You need to give us more detail about your data and some summary statistics and plots for anyone to get any further with this. Edit them into the question please not as comments. $\endgroup$ – mdewey Oct 9 '16 at 12:42
  • $\begingroup$ Please register &/or merge your accounts (you can find information on how to do this in the My Account section of our help center), then you will be able to edit & comment on your own question. $\endgroup$ – gung Oct 9 '16 at 17:44
2
$\begingroup$

To answer your question: Technically yes, the Method 1 model is better than the Method 2 model. It generates a higher probability of ranking positive instance higher than a negative instance - meaning that there's a higher chance that it actually classifies correct.

The problem is, as godspeed mentions, that anything below 0.5 is worse than classifying by flipping a coin, so both models actually suck - and it could be a sign of something going wrong in your model building.

$\endgroup$
1
$\begingroup$

Consider that if you choose the opposite of the prediction made by both classifiers, your AUC becomes 0.80 and 0.85 respectively. So my suggestion would be to check the labels on both training and test data to ensure they are not switched for some reason. Otherwise, both models should be considered "bad" as is, and the one with the smallest AUC as even worse

$\endgroup$
  • $\begingroup$ The labels are correct, it is just that the size of features used in both methods is very small. Can I make a conclusion that the first method works better in such a context? $\endgroup$ – bumblbee Oct 9 '16 at 17:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.