0
$\begingroup$

Most modern object detection algorithms rely on neural networks and output a bounding box and confidence for each object (or more accurately, a confidence for each possible object class considered, the highest of which is taken to be the true object confidence and to define the predicted class for that object). Beyond giving a relative likelihood of "objectness" (a predicted bounding box with 0.9 confidence is more likely to actually contain an object than a predicted bounding box with 0.1 confidence), is there any statistical significance to these predictions? For instance, should we be able to say that a predicted confidence of 0.1 indicates that roughly 10% of predictions at this confidence are actually objects? Is there another statistically meaningful definition?

I understand that at best this would be a loose statistical meaning because neural networks would be trained with loss functions that codify this statistical distribution into a loss, and the network would then only approximate this statistical distribution. Thus the distribution would also depend on the loss function used to train the classification portion of the neural network. These issues notwithstanding, is there or was there at some point inherent meaning to the confidence measure, or is it purely a relative quantity?

$\endgroup$
0
$\begingroup$

The problem you're describing is known as "model calibration". A model is well calibrated if the predicted the score corresponds to the expected probability. Simple methods like logistic regression tend to produce well calibrated models, whereas some of the tricks used in modern neural networks tend to produce poorly calibrated models. This paper suggests that Temperature Scaling is a simple and effective way to calibrate modern neural nets: https://arxiv.org/abs/1706.04599

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.