I completely understand that it's ok to use a loss function different from the evaluation metric. For example, accuracy isn't computationally feasible to optimize directly since it's not differentiable. So instead I could choose to minimize, say, the hinge loss plus regularization in a soft SVM, in the hope that the result will also have a decent accuracy.

However, I don't understand how this argument carries over to the (more common) case when the evaluation metric isn't symmetric in the classes.

Depending on the cost of each type of mis-classification, and the baseline probability of each class, we normally choose an appropriate evaluation metric (something like expected cost of error), which usually puts greater emphasis on one type of mistake than the other (otherwise we'd never look at ROC or PR curves, and just use accuracy every time). However, the training process is usually based on a loss function that is symmetric in the classes (e.g., cross-entropy or hinge loss or MSE etc. plus some regularization terms).

Example: in a binary classification case, suppose the baseline probabilities are the same, but type I error is much more costly than type II error. So I choose an evaluation metric that heavily penalizes false positives. However, when I train a learner (be it ANN or RF or SVM or logistic regression), the training process is completely oblivious to this fact, and tries equally hard to fit each example, regardless of whether the target class is positive or negative. Sure, after the training finishes, I get to choose the threshold for converting the numeric output into a class label. But this is just a single degree of freedom out of the dozens or hundreds degrees of freedom that the learner had during training. I would think the choice of the threshold, while helpeful, can't possibly compensate for the inconsistent incentive provided to the learner during the training.

What am I missing? Why, in the standard machine learning practice, we don't modify the loss functions (e.g., by weighting) to better reflect the evaluation metric?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.