Timeline for Why to choose AUC over accuracy?
Current License: CC BY-SA 4.0
15 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 2, 2019 at 13:59 | comment | added | David | Let us continue this discussion in chat. | |
| Jul 2, 2019 at 13:58 | comment | added | David | @TamasFerenci No. Presenting accuracy as a validation measure does not mean you know the cost function, because the fact that you use accuracy does not mean that you use ONLY accuracy. Different types of measures such as precision, recall, F1-score, accuracy and many others can be used simultaneously for model validation. Even when the cost is known, you should not leave the entire model-building/validating process to how it performs on one single metric | |
| Jul 2, 2019 at 13:58 | comment | added | Tamas Ferenci | "every diagnose test should return 100% positives if we attend to the cost function" No, that's absolutely not true. False positivity also has a cost attached (possibly more invasive or risky further diagnostic or treatment procedures, psychological stress of being diagnosed with an illness, monetary costs of further diagnostics etc.). | |
| Jul 2, 2019 at 13:56 | comment | added | Tamas Ferenci | If you don't know something, the best is to say that you don't know. In this case: if you have no information on the costs, you can run a sensitivity analysis, presenting results for different cost structures. It is very important to note that using accuracy essentially means that you say you know the cost structure, while your very starting point was just that you don't know it! In other words, the usage of accuracy will imply that you consider something to be known perfectly, while your starting point was just the total opposite, that it is not even possible to formalize in your problem. | |
| Jul 2, 2019 at 13:49 | comment | added | David | @TamasFerenci What cost function should I use then when I don't what the cost function is? It is true that accuracy can be though of as a particular type of cost function, but it has a meaning beyond that (overall proportion of times you screw up) Cost functions present additional problems: for example, if unless we want to give a finite value in cash to human life, every diagnose test should return 100% positives if we attend to the cost function | |
| Jul 2, 2019 at 13:46 | comment | added | Tamas Ferenci | That's indeed a problem, but using accuracy has nothing to do with this, as accuracy also implies a cost structure. I mean, accuracy is a cost-based decision, with one particular cost function, implicitly defined. If it is not even possible to formalize the cost function in your problem, then what makes you think that this particular one will be correct...? | |
| Jul 2, 2019 at 13:40 | comment | added | David | @TamasFerenci I disagree, since slight changes in "cost" will force you to retrain the entire model all over again. It is also not always possible to formalize a cost function in mathematical terms | |
| Jul 2, 2019 at 13:32 | comment | added | Tamas Ferenci | You're still better off with the (correct) cost-based approach, as it work in that case as well (just set the costs equal, no problem here!), in contrast to accuracy which works only under special circumstances. | |
| Jul 2, 2019 at 13:10 | comment | added | David | @TamasFerenci But that will hardly ever happen in the case I presented, where the errors will pretty much all the time be symmetric. There are also disadvantages in manually setting costs into your cost function | |
| Jul 2, 2019 at 12:21 | comment | added | Tamas Ferenci | I can! If you really-really-really don't want to ever think of a sender of being female when he is indeed male (but you have much less problem with predicting a female sender as male), then you might very well prefer a model that never does the former error, and does the latter in 10% of the cases (accuracy: 90%), then one which never commits the latter, but commits the former in 3% of the cases (accuracy: 97%). | |
| Jul 2, 2019 at 11:56 | comment | added | David | @TamasFerenci Why not? If men and women write me messages at a similar rate and I want to predict whether the last anonymous message I received comes from a male or female, I cannot think of a single sitaution where getting the right answer 97% of the time is not enough of a reason to feel happy about my work! | |
| Jul 2, 2019 at 11:46 | comment | added | Tamas Ferenci | You're right, sorry, I didn't get "asymmetrical" at first glance. Nevertheless, I suggest not calling accuracy a "legitimate metric" in any case. Sorry again! | |
| Jul 2, 2019 at 11:44 | comment | added | David | @TamasFerenci That's literally what is being stated on the very next sentence. "Also, errors are rarelly symetrical (for instance, in medicine, false positives and false negatives are not the same)" | |
| Jul 2, 2019 at 11:21 | comment | added | Tamas Ferenci | "Accuracy is a legitimate validation metric when you are working with a balanced dataset." The problem is, that there is another, completely different issue: Accuracy doesn't take costs into account. See many articles of Frank Harrell, or my answer here: stats.stackexchange.com/questions/368949/… . | |
| Jul 2, 2019 at 11:15 | history | answered | David | CC BY-SA 4.0 |