10
$\begingroup$

I'm currently working on a project involving using different sets of data as a predictor to predict the outcome of out-sample data. I use AUC (Area under the Curve of ROC) to compare the performances of each set of data.

I am familiar with the theory behind AUC and ROC, but I'm wondering is there a precise standard for assessing AUC, for example, if an AUC outcome is over 0.75, it will be classified as a 'GOOD AUC', or below 0.55, it will be classified as a 'BAD AUC'.

Is there such a standard, or AUC is always for comparing only?

$\endgroup$
2
  • 32
    $\begingroup$ If you are a trader and you can get an AUC of 0.501 in predicting future financial transactions, you're the richest man in the world. If you are a CPU engineer and your design gets an AUC of 0.999 at telling if a bit is 0 or 1, you have a useless piece of silicon. $\endgroup$ – Calimo Aug 16 '20 at 7:39
  • 1
    $\begingroup$ No, there is no general standard. But maybe a standard can be made for your specific project. The question should not be 'What is the standard' but instead 'How do I make a standard (for my specific job/project)'. In order for that to be answered you need to formulate your project more accurately and formalize the problem (e.g. what it the point of the classifications, what context or measurements do you have to compare, what is the cost of a misclasification and what is the gain of a good classification, etc.). Currently it is too broad. $\endgroup$ – Sextus Empiricus Aug 16 '20 at 23:49
18
$\begingroup$

From the comments:

Calimo: If you are a trader and you can get an AUC of 0.501 in predicting future financial transactions, you're the richest man in the world. If you are a CPU engineer and your design gets an AUC of 0.999 at telling if a bit is 0 or 1, you have a useless piece of silicon.

$\endgroup$
10
$\begingroup$

This is a complementary to Andrey's answer (+1).

When looking for a generally accepted reference on AUC-ROC values, I came across Hosmer's "Applied Logistic Regression". In Chapt. 5 "Assessing the Fit of the Model", it emphasised that "there is no “magic” number, only general guidelines". Therein, the following values are given:

  • ROC = 0.5 This suggests no discrimination, (...).
  • 0.5 < ROC < 0.7 We consider this poor discrimination, (...).
  • 0.7 $\leq$ ROC < 0.8 We consider this acceptable discrimination.
  • 0.8 $\leq$ ROC < 0.9 We consider this excellent discrimination.
  • ROC $\geq$ 0.9 We consider this outstanding discrimination.

These values are by no means set-to-stone and they are given without any context. As Star Trek teaches us: "Universal law is for lackeys, context is for kings", i.e. (and more seriously) we need to understand what we are making a particular decision and what our metrics reflect. My guidelines would be:

  1. For any new task we should actively look at existing literature to see what is considered competitive performance. (e.g. detection of lung cancer from X-ray images) This is practically a literature review.
  2. If our tasks is not present in literature, we should aim to provide an improvement over a reasonable baseline model. That baseline model might be some simple rules of thumb, other existing solutions and/or predictions provided by human rater(s).
  3. If we have a task with no existing literature and no simple baseline model available, we should stop trying to make a "better/worse" model performance comparison. At this point, saying "AUC-R0C 0.75 is bad" or "AUC-ROC 0.75 is good" is a matter of opinion.
$\endgroup$
8
$\begingroup$

It isn't possible to say because it really depends on the task and the data. For some simple tasks AUC can be 90+, for others ~0.5-0.6.

$\endgroup$
4
  • 3
    $\begingroup$ Correct. +1. But I would note that an AUC-ROC that is below ~0.65 is indicative of a weak classifier. Hosmer's "Applied Logistic Regression" suggest 0.70 as rule of thumb for the lower threshold of "acceptable discrimination". $\endgroup$ – usεr11852 Aug 16 '20 at 9:19
  • 4
    $\begingroup$ In general this is true, but there are cases, when it isn't really possible to get high AUC due to the task formulation. For example, I had a project, where it was necessary to predict a probability of customer buying a certain model of smartphone next month. This isn't really predictable, so AUC was lower than 0.6. So any model on such a task would be weak. $\endgroup$ – Andrey Lukyanenko Aug 16 '20 at 11:41
  • 3
    $\begingroup$ Fully agreed. The will for a high AUC (or any other metric for that matter) does not necessitate it's achievable. $\endgroup$ – usεr11852 Aug 16 '20 at 12:37
  • 2
    $\begingroup$ I think it is still vital to understand an AUC of around 0.5 is not a good sign. It is mentioned here as an option for some tasks, but, essentially this means the model just randomly predicts. $\endgroup$ – Thomas Aug 16 '20 at 14:07
3
$\begingroup$

Generally, I would not say so. It all depends on the task, your data set, and objectives. There is no rule of thumb that an AUC value of x.x is defined as a good predicting model.

That being said, you want to achieve as high an AUC value as possible. In cases where you get an AUC of 1, your model is essentially a perfect predictor for your outcome. In cases of 0.5, your model is not really valuable. An AUC of 0.5 just means the model is just randomly predicting the outcome no better than a monkey would do (in theory). I can only recommend you to read more about it if you have not so. This is realtively straightforward. And, here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy