Is accuracy a reliable evaluation metric for training a classification model? [duplicate]

Question

What is the problem in training on accuracy when balancing classes during training? I Understand why accuracy is not a good metric for the test data evaluation of the model, as in the test data we have no control over class imbalance. This should be the same reason why F1/Dice score is bad for evaluating classification in the test set (although currently used in most segmentation challenges -BRATS, ISLES,...-).

But I see no problem in optimizing on accuracy over training samples, for which we can control class imbalance through selective sampling.

Is there any reason not to use accuracy as the training metric, while being careful not to use it for evaluation of generalization?

Answer 1

The issues with accuracy are well documented.

Practically: It almost certainly does not measure what you really care about. Accuracy assigns equal severity to all errors in classification. This is almost always unrealistic.

Statistically: It fails to be sensitive to the probabilistic nature of classification. It is a ridiculous delusion of grandeur to believe that a predictive model should be able to preform hard class assignments, that's not how the universe works. All of our learnings from data are uncertain, and it is a careful scientist's job to quantify their uncertainty.

Mixing of Concerns: Assigning classes and taking actions is a business, not a statistical, problem. A statistical model's job is to estimate predicted probabilities, which you may then use as an input to a decision rule to solve a business or scientific problem. This has many benefits, including a clean separation of concerns, an adaptability of the model to different problem domains, and an ability to estimate the expected benefits and costs of various successes or errors in decision making.

Mathematically: The accuracy loss function is not even continuous, so in particular it is not differentiable, and very not-smooth. Optimization techniques work best on smooth functions, so it's unclear how one would fit such a model.

Culturally: A focus on accuracy in introductory ML and statistical learning courses creates cultural problems. Observe the backlog of questions on this site evaluating their logistic regressions by naively thresholding at a 0.5 predicted probability, getting poor results in terms of accuracy, and wondering what's wrong with their model. Nothing is wrong with their model, logistic regression's job is to estimate conditional probabilities, which has been implicitly rejected.

None of this has anything to do with class balance. The issue with accuracy arising from class balance is real, but secondary to deeper problems.

Answer 2

The accuracy measure treats all misclassifications the same. Sometimes this is fine, but other times, false positives and false negatives cannot be treated equally. For example, in the domain of medical diagnosis, false negatives are much worse than false positives, because leaving a serious disease untreated is much less desirable than some additional testing that turns out to be unnecessary. You can try to fudge the training step by skewing the class imbalance to resemble your relative misclassification cost, but it's generally better to just use a metric that lets you weight the errors differently to begin with.