Skip to main content
deleted 2 characters in body
Source Link
Sycorax
  • 94.1k
  • 23
  • 236
  • 390

Accuracy is not a great way to displacereport machine learning results. (I've never found a need to report accuracy, except when explaining my results to a non-technical audience.) Accuracy only compares a predicted score $t$ to some cutoff $c$, which is not a proper scoring rule and conceals important information about model fitness.

I assume you're using some sort of proper loss function in the "loss" graph, such as cross-entropy loss. Cross-entropy loss is more useful than accuracy because it is sensitive to "how wrong" its results are: if the label is $1$ but $t=0.9$, the cross-entropy is lower than when the label is $1$ but $t=0.1$.

The phenomenon you're seeing when comparing these two graphs -- accuracy is flat but loss is increasing -- happens because $t>c$ is satisfied in the accuracy graph, but the predicted scores are poorly aligned to their labels.

This is intimately related to this similar issue with AUC: Why is AUC higher for a classifier that is less accurate than for one that is more accurate?

Accuracy is not a great way to displace machine learning results. (I've never found a need to report accuracy, except when explaining my results to a non-technical audience.) Accuracy only compares a predicted score $t$ to some cutoff $c$, which is not a proper scoring rule and conceals important information about model fitness.

I assume you're using some sort of proper loss function in the "loss" graph, such as cross-entropy loss. Cross-entropy loss is more useful than accuracy because it is sensitive to "how wrong" its results are: if the label is $1$ but $t=0.9$, the cross-entropy is lower than when the label is $1$ but $t=0.1$.

The phenomenon you're seeing when comparing these two graphs -- accuracy is flat but loss is increasing -- happens because $t>c$ is satisfied in the accuracy graph, but the predicted scores are poorly aligned to their labels.

This is intimately related to this similar issue with AUC: Why is AUC higher for a classifier that is less accurate than for one that is more accurate?

Accuracy is not a great way to report machine learning results. (I've never found a need to report accuracy, except when explaining my results to a non-technical audience.) Accuracy only compares a predicted score $t$ to some cutoff $c$, which is not a proper scoring rule and conceals important information about model fitness.

I assume you're using some sort of proper loss function in the "loss" graph, such as cross-entropy loss. Cross-entropy loss is more useful than accuracy because it is sensitive to "how wrong" its results are: if the label is $1$ but $t=0.9$, the cross-entropy is lower than when the label is $1$ but $t=0.1$.

The phenomenon you're seeing when comparing these two graphs -- accuracy is flat but loss is increasing -- happens because $t>c$ is satisfied in the accuracy graph, but the predicted scores are poorly aligned to their labels.

This is intimately related to this similar issue with AUC: Why is AUC higher for a classifier that is less accurate than for one that is more accurate?

Source Link
Sycorax
  • 94.1k
  • 23
  • 236
  • 390

Accuracy is not a great way to displace machine learning results. (I've never found a need to report accuracy, except when explaining my results to a non-technical audience.) Accuracy only compares a predicted score $t$ to some cutoff $c$, which is not a proper scoring rule and conceals important information about model fitness.

I assume you're using some sort of proper loss function in the "loss" graph, such as cross-entropy loss. Cross-entropy loss is more useful than accuracy because it is sensitive to "how wrong" its results are: if the label is $1$ but $t=0.9$, the cross-entropy is lower than when the label is $1$ but $t=0.1$.

The phenomenon you're seeing when comparing these two graphs -- accuracy is flat but loss is increasing -- happens because $t>c$ is satisfied in the accuracy graph, but the predicted scores are poorly aligned to their labels.

This is intimately related to this similar issue with AUC: Why is AUC higher for a classifier that is less accurate than for one that is more accurate?