6
$\begingroup$

If a set of features, A, results in an AUC of 0.5, then an improved set of features, B, were used resulting in an AUC of 0.75, how do I express this improvement in words:

  1. 50% improvement (0.75-0.5)?
  2. 25% improvement (0.75-0.5)/0.5?

it seems like simple statistics, but I am mixed up on this issue.

Improve this question
$\endgroup$
1
  • $\begingroup$ The % numbers seem to be with the wrong expressions in the list. $\endgroup$
    – GeoMatt22
    Aug 27, 2016 at 5:07

3 Answers 3

Reset to default
3
$\begingroup$

You can equivalently use the following expressions:

Speaking of correct terms, AUROC (if that is your metric) should be used instead of AUC, as the latter term is ambiguous.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Your second measure of improvement is called kappa. $\endgroup$
    – meh
    Aug 26, 2016 at 21:44
2
$\begingroup$

I just posted a somewhat related question here: Compare and quantify relative improvement in ROC AUC scores? that has this as a component to it.

I'm not sure, though given the ROC AUC represents the probability that a classifier scores a positive case higher than a negative case, perhaps comparing the magnitude of the odds ratios would be an appropriate method for comparison.

So rather than 0.75 / 0.50, this becomes (3/1) / (1/1), so the improvement is 3x, i.e. the new model has 3x the odds of scoring a positive case more highly than a negative case compared to the old model.

(To the point above, 0.50 represents random chance, so perhaps these should be rescaled to between 0.5 and 1 which changes the interpretation though. Also in the case when you're comparing against a model with ROC AUC of 0.5, any improvement would represent infinite improvement. Comparing the odds ratios under this rescaled version also means that an improvement from 0.51 to 0.59 would represent the same amount of improvement as 0.91 to 0.99, however the former may feel more likely to arise randomly -- so would probably want to also consider some notion of confidence when comparing ROC AUCs.)

Improve this answer
$\endgroup$
0
$\begingroup$

To calculate your relative improvement assumed to be $x$, you should use $x=(0.75-0.5)/0.5$.

I believe the relative improvement is more intuitive in the AUC case. Also, AUC is commomly used in literature, so both AUC and AUROC arr okay

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I don't think we should encourage the use of ambiguous terms :-) $\endgroup$
    – Franck Dernoncourt
    Aug 27, 2016 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.