4
$\begingroup$

To the best of my knowledge, cross entropy is consistent with MLE (maximum likelihood estimation), assume we have data with binomial distribution, if we do MLE on the data, then our loss function would be just the cross entropy function.

The consistency between cross entropy and MLE indicates that if we could find the optimization with regard to the cross entropy function, then we theoretically solved the optimization problem using MLE.

However, the focal loss is kind of weighted cross entropy, which would probably not be consistent with MLE. Theoretically MLE would give us the best parameters, then why models using focal loss which is not consistent with MLE could obtain better performances in contrast with models using cross entropy?

$\endgroup$
0
1
$\begingroup$

There are multiple concepts in this question. In statistics, you will find there are theorems for certain models about what methods produces the "best" parameter estimates. In linear regression with normally distributed errors, you can prove that the normal statistical parameter estimates are the BLUE (Best linear unbiased estimates) of the regression parameters. In other contexts, you find MLE (maximum likelihood estimates) and MVUE (minimum variance unbiased estimates) are the best we can do statistically. When you cross into machine learning and start to use a loss function for an optimization, you are stepping away from the statistical concept of what is "best" and instead thinking about the parameters that produce the minimum cost/loss (in training, validation, or test). Further, what makes a good loss function in machine learning is also related to guarding against overfitting. The MLE doesn't have any such considerations. Once the likelihood function is specified, there isn't any consideration for overfitting, you are only attempting to find the parameters that maximize the likelihood.

$\endgroup$
1
+50
$\begingroup$

The fact that cross entropy has an interpretation as a probability distribution or as a the Kullback-Leibler divergence between's the model estimated probability distribution and the empirical distribution of labels does not mean that it is optimal for all cases. As you point out, classification with focal loss actually works even better.

The original paper (Focal Loss for Dense Object Detection) explains the motivation behind it in great detail, and why it is expected to yield better performance. Maybe most relevant for the motivation of your question, it has already been proven that focal loss leads to better calibrated networks (see Calibrating Deep Neural Networks using Focal Loss). And well calibrated classifiers can actually have a better performance (Properties and benefits of calibrated classifiers).

Notice that MLE has problems in situations like imbalanced datasets. That has motivated the development of resampling strategies to compensate for it. Focal loss can be seen as a more principled way to weight samples.

Calibration is about how correct your predictions are, on average. As an example, say you want to predict whether tomorrow is going to rain or not. If your perfectly calibrated model tells you it is going to rain with 70% chance, it means that, every time you get such a score from your estimator, 70% of the times it will actually rain. This is specially relevant for decision making. This way you can estimate what your average outcomes (gain/losses) will be.

$\endgroup$
2
  • $\begingroup$ Could you elaborate on what calibrated means intuitively? BTW, thanks for good resources, bo no time to read them thoroughly :shrug: $\endgroup$
    – hans
    Feb 9 at 6:14
  • $\begingroup$ just extended my answer $\endgroup$
    – jpmuc
    Feb 9 at 19:31
0
$\begingroup$

You have to define "best parameters" and "better performance". Normally, you choose a loss function that's differentiable, but you actually care about some other metric, like accuracy, AUC, or average precision. If, for example, you care about AUC, and using focal loss produces higher validation AUC than cross entropy loss does, then use focal loss. In other words, the loss function you end up using is a (discrete) hyperparameter you tune to optimize the metric you care about.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.