why focal loss works

Question

To the best of my knowledge, cross entropy is consistent with MLE (maximum likelihood estimation), assume we have data with binomial distribution, if we do MLE on the data, then our loss function would be just the cross entropy function.

The consistency between cross entropy and MLE indicates that if we could find the optimization with regard to the cross entropy function, then we theoretically solved the optimization problem using MLE.

However, the focal loss is kind of weighted cross entropy, which would probably not be consistent with MLE. Theoretically MLE would give us the best parameters, then why models using focal loss which is not consistent with MLE could obtain better performances in contrast with models using cross entropy?

Answer 1

The fact that cross entropy has an interpretation as a probability distribution or as a the Kullback-Leibler divergence between's the model estimated probability distribution and the empirical distribution of labels does not mean that it is optimal for all cases. As you point out, classification with focal loss actually works even better.

The original paper (Focal Loss for Dense Object Detection) explains the motivation behind it in great detail, and why it is expected to yield better performance. Maybe most relevant for the motivation of your question, it has already been proven that focal loss leads to better calibrated networks (see Calibrating Deep Neural Networks using Focal Loss). And well calibrated classifiers can actually have a better performance (Properties and benefits of calibrated classifiers).

Notice that MLE has problems in situations like imbalanced datasets. That has motivated the development of resampling strategies to compensate for it. Focal loss can be seen as a more principled way to weight samples.

Calibration is about how correct your predictions are, on average. As an example, say you want to predict whether tomorrow is going to rain or not. If your perfectly calibrated model tells you it is going to rain with 70% chance, it means that, every time you get such a score from your estimator, 70% of the times it will actually rain. This is specially relevant for decision making. This way you can estimate what your average outcomes (gain/losses) will be.

Answer 2

There are multiple concepts in this question. In statistics, you will find there are theorems for certain models about what methods produces the "best" parameter estimates. In linear regression with normally distributed errors, you can prove that the normal statistical parameter estimates are the BLUE (Best linear unbiased estimates) of the regression parameters. In other contexts, you find MLE (maximum likelihood estimates) and MVUE (minimum variance unbiased estimates) are the best we can do statistically. When you cross into machine learning and start to use a loss function for an optimization, you are stepping away from the statistical concept of what is "best" and instead thinking about the parameters that produce the minimum cost/loss (in training, validation, or test). Further, what makes a good loss function in machine learning is also related to guarding against overfitting. The MLE doesn't have any such considerations. Once the likelihood function is specified, there isn't any consideration for overfitting, you are only attempting to find the parameters that maximize the likelihood.

Answer 3

You have to define "best parameters" and "better performance". Normally, you choose a loss function that's differentiable, but you actually care about some other metric, like accuracy, AUC, or average precision. If, for example, you care about AUC, and using focal loss produces higher validation AUC than cross entropy loss does, then use focal loss. In other words, the loss function you end up using is a (discrete) hyperparameter you tune to optimize the metric you care about.

Answer 4

As @rcarnell explains, MLE (maximum likelihood estimation) is optimising on the training set. I would add that it has traditionally been performed on purposely low complexity models (eg simple low parameter linear models), and so are not at risk of overfitting.

Machine Learning uses models with very many parameters, for which overfitting is an issue. In this case regularised regression (eg L2 regularisation/weight decay/stopped training) will perform better than unregularised MLE for predicting new samples (generalisation)

Intuitively I prefer loss functions that are a sum of an error term and a complexity term. Focal loss is a mix of these hoping to optimise test NLL (or perhaps rather accuracy and calibration), in much the same way as regularised regression does (by not optimising just training set negative log likelihood, NLL). In the paper Calibrating Deep Neural Networks using Focal Loss they suggest focal loss is an upper bound of such an additive loss:( NLL - $\gamma$ entropy(predictions)) [we want to minimise NLL and maximise entropy].

As mentioned in that paper,

The promising performance of weight decay Guo et al., 2017 (regulating the norm of weights) on the calibration of neural networks can perhaps be explained using this. This increase in the network’s confidence during training is one of the key causes of miscalibration.

You have this generalisation problem even in simple logistic regression, in particular, you can view 'perfect separation', where MLE on small amounts of data causes the weights to 'blow up' and the probability estimates to go to 0/1 as an overfitting problem. The standard solution there is to use regularised logistic regression (eg L2, which is equivalent to weight decay). Infact in the calibrating deep neural networks paper, they show that focal loss also works for this case in Appendix C.

In Guo et al., 2017 they point out that weight decay has gone out of fashion, despite the fact that it seems to help calibration error. Whilst I could see it's not so helpful for all the relu hidden layers, it would seem to be beneficial for the sigmoid output layer.