1
$\begingroup$

I have a binary classification problem, where there are multiple correct predictions, however, I would consider the prediction to be correct if the highest confidence prediction of a 1 is correct.

I have had limited success with a CNN using the mean square error loss function. I think the reason is that, this loss function is calculating the loss between every element in y_true and y_pred.

Example:

I want my model to confidently predict one of these cells as a 1. There probably isn't enough information in the input data to accurate predict every 1, especially in the areas far from input cells with a positive value, therefore I would be happy if the model is conservative and predicts a 0 where it is unsure.

input_data

enter image description here

y_pred

enter image description here

I believe the model is being optimized to classify as many cells correctly as possible.

It's not possible to correctly predict the cell in the bottom right corner in this example. Therefore the model should take the conservative approach and predict a 0.

y_true

enter image description here

A perfect loss function in my mind would:

Heavily penalize the case where (y_true, y_pred) = (0,1) (false positive)

Heavily reward the case where (y_true, y_pred) = (1,1) (true positive)

Moderately penalize the cases where (y_true, y_pred) = (1,0) (false negative)

Moderately reward the cases where (y_true, y_pred) = (0,0) (true negative)

Or

Calculate the loss only on the cell with the highest value in y_pred

Can anybody suggest a suitable loss function?

Let me know if I have omitted any required detail

$\endgroup$

1 Answer 1

0
$\begingroup$

The mean squared loss is a proper scoring rule, also known as the Brier score. That is, it will be optimized by probabilistic predictions that reflect the true probability of a cell being of the target class. There are a few other proper scoring rules you could consider, e.g., the log score. Here is a comparison between the two: Why is LogLoss preferred over other proper scoring rules?

It makes no sense to require a higher confidence (i.e., a higher predicted probability) if there is no reason to have a higher confidence per the data. If your probability of having COVID (e.g., based on symptoms) is 0.6, why would you want a predictive algorithm to output a biased probabilistic prediction of 0.9? Thus, proper scoring rules are precisely what you want.

Separate from the probabilistic prediction is the question of a hard decision. If you need this (and I maintain that it makes more sense to separate the two aspects), then you can compare your probabilistic prediction to a threshold. In the COVID example, you might take certain actions (e.g., wearing a mask to protect others) based on one threshold (e.g., if your probabilistic prediction of having COVID exceeds 0.1), and take other actions based on a different threshold (e.g., take time off from work to get a PCR test at the doctor, but only if your probability exceeds 0.3). In tuning thresholds, you should take the cost of decisions in the light of the true outcome into account, and here you can use a high threshold to only treat a cell as positive it has a high predicted probability. See Reduce Classification Probability Threshold.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.