# Disadvantages of the Kullback-Leibler divergence

I'm working on a calibration problem which involves the usage of the Kullback-Leibler divergence as an error between some empirical distribution $p$ and a theoretical distribution $q$. In the model, the $q$ distribution is normal with some fixed parameters. I have two questions:

1. Is the Kullback-Leibler divergence the best f-divergence to consider as error?
2. Does the usage of the Kullback-Leibler divergence entail any kind of issue?
• I think supplying some more information about the problem you are considering would be helpful. Otherwise, the only really authoritative answer to either of your two question is: "It depends", which I imagine you will find unsatisfying. :-) – cardinal Dec 3 '13 at 0:25
• If you are interested in the (expected) loglikelihood ratio between the two distribution: stats.stackexchange.com/questions/188903/… then KL divergence can be relevant. Otherwise, maybe not. – kjetil b halvorsen May 25 '17 at 17:30

I'd like to add the first answer, which would be unsatisfying, to this question through the lens of deep learning mostly in NLP:

First things first,

Disadvantages of the Kullback-Leibler divergence

Let's see the definition (in terms of your question):
$$KL(q||p)=\sum q(s)\log \frac{q(s)}{p(s)}$$ When $p(s) > 0$ and $q(s)\to 0$, the KL divergence shrinks to 0, which means MLE assigns an extremely low cost to the scenarios, where the model generates some samples that do not locate on the data distribution.

Consider this, the corpus in hand includes the whole samples existing in the world then $q(s) \to 0$ indicates that $s$ occurs very rarely in the corpus (the law of large number), and then its probability may happen to be very large (due to samples lookalike but different or opposite in fact). In this case, because of the lack of training for this kind of category and hence its high probabilities in the distribution, such rare samples that do not locate on the data distribution may be generated while we are testing or validating.