Timeline for Overfitting on the loss graph, but not the accuracy graph
Current License: CC BY-SA 3.0
11 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 28, 2017 at 23:41 | vote | accept | Yuri | ||
| Aug 28, 2017 at 22:40 | comment | added | Sycorax♦ | I can't speak for what someone else did but if their analysis begins and ends with comparisons of accuracy, the model is bogus. | |
| Aug 28, 2017 at 22:13 | comment | added | Yuri | Also talking about reporting accuracy. For example, in this paper arxiv.org/abs/1408.5882 the author reports exactly accuracy. Or you are saying he checked his model for overfitting using the cross entropy and then reported only accuracy? | |
| Aug 28, 2017 at 21:50 | vote | accept | Yuri | ||
| Aug 28, 2017 at 22:37 | |||||
| Aug 28, 2017 at 19:18 | comment | added | Sycorax♦ | All of that is true. But it's also true that Brier scores and log-likelihood don't report the same information as ROC curves. Hammers aren't screwdrivers. | |
| Aug 28, 2017 at 19:17 | comment | added | Yuri | But the first comment under the link you provided says: "I wish I had a good reference for that, but briefly any measure based solely on ranks such as cc (AUROC) cannot give enough credit to extreme predictions that are "correct". Brier, and even more so the logarithmic scoring rule (log likelihood) give such credit. This is also an explanation why comparing two cc-indexes is not competitive with other approaches power-wise" | |
| Aug 28, 2017 at 18:50 | vote | accept | Yuri | ||
| Aug 28, 2017 at 18:50 | |||||
| Aug 28, 2017 at 18:48 | history | edited | Sycorax♦ | CC BY-SA 3.0 |
deleted 2 characters in body
|
| Aug 28, 2017 at 18:44 | comment | added | Sycorax♦ | Quite the opposite -- AUROC is a measurement based on the relative ranks of the scores for positives and negatives, specifically the probability that a random positive is ranked higher than a random negative. This is sensitive, indirectly, to the alignments of scores and labels, and "how wrong" the classifier is on average. Moreover, the operating points give information about tradeoffs of errors are different cutoffs. | |
| Aug 28, 2017 at 18:42 | comment | added | Yuri | Thank you very much for the answer. I was going to ask about AUC, but you already answered my question. So AUC is not a good measure too? | |
| Aug 28, 2017 at 18:39 | history | answered | Sycorax♦ | CC BY-SA 3.0 |