Both of Cross-entropy and KL divergence are tools to measure the distance between two probability distribution. What is the difference? $$ H(P,Q) = -\sum_x P(x)\log Q(x) $$ $$ KL(P | Q) = \sum_{x} P(x)\log {\frac{P(x)}{Q(x)}} $$ Moreover, minimization of KL is equivalent to minimization of Cross-Entropy.

I want to know them instinctively.

Thank you very much in advance.

  • $\begingroup$ You're missing a $\ln$ in the KL definition. I will add it in, but noted here so other can identify a source of confusion. $\endgroup$ – conjectures Jul 19 '18 at 13:18

You will need some conditions for claiming the equivalence of minimizing cross entropy to minimizing KL divergence. I will put your question under the context of classification problems using cross entropy as loss functions.

Let us first recall that entropy is used to measure the uncertainty of a system, which is defined as \begin{equation} S(v)=-\sum_ip(v_i)\log p(v_i)\label{eq:entropy}, \end{equation} for $p(v_i)$ as the probabilities of different states $v_i$ of the system. From an information theory point of view, $S(v)$ is the amount of information is needed for removing the uncertainty.

For instance, the event A I will die eventually is almost certain (maybe we can solving the aging problem for word almost), therefore it has low entropy which requires only the information of the aging problem cannot be solved to make it certain. However, the event B The president will die in 50 years is much more uncertain than A, thus it needs more information to remove the uncertainties.

Now look at the definition of KL divergence between events A and B \begin{equation} D_{KL}(A\parallel B) = \sum_ip_A(v_i)\log p_A(v_i) - p_A(v_i)\log p_B(v_i)\label{eq:kld}, \end{equation} where the first term of the right hand side is the entropy of event A, the second term can be interpreted as the expectation of event B in terms of event A. And the $D_{KL}$ describes how different B is from A from the perspective of A.

To relate cross entropy to entropy and KL divergence, we formalize the cross entropy in terms of events A and B as \begin{equation} H(A, B) = -\sum_ip_A(v_i)\log p_B(v_i)\label{eq:crossentropy}. \end{equation} From the definitions, we can easily see \begin{equation} H(A, B) = D_{KL}(A\parallel B)+S_A\label{eq:entropyrelation}. \end{equation} If $S_A$ is a constant, then minimizing $H(A, B)$ is equivalent to minimizing $D_{KL}(A\parallel B)$.

A further question follows naturally as how the entropy can be a constant. In a machine learning task, we start with a dataset (denoted as $P(\mathcal D)$) which represent the problem to be solved, and the learning purpose is to make the model estimated distribution (denoted as $P(model)$) as close as possible to true distribution of the problem (denoted as $P(truth)$). $P(truth)$ is unknown and represented by $P(\mathcal D)$. Therefore in an ideal world, we expect \begin{equation} P(model)\approx P(\mathcal D) \approx P(truth) \end{equation} and minimize $D_{KL}(P(\mathcal D)\parallel P(model))$. And luckily, in practice $\mathcal D$ is given, which means its entropy $S(D)$ is fixed as a constant.

  • $\begingroup$ Thank you for your answer. It deepened my understanding. So when we have a dataset, it is more effective to minimize cross- entropy rather than KL, right? However, I cannot understand the proper use of them. In other words, when should I minimize KL or cross entropy? $\endgroup$ – Jourd Jul 19 '18 at 14:00
  • $\begingroup$ After reading your answer, I think it is no use to minimize KL because we always have a dataset, P(D). $\endgroup$ – Jourd Jul 19 '18 at 14:03
  • $\begingroup$ Ideally, one would choose KL divergence to measure the distance between two distributions. In the context of classification, the cross-entropy loss usually arises from the negative log likelihood, for example, when you choose Bernoulli distribution to model your data. $\endgroup$ – doubllle Jul 19 '18 at 14:14
  • $\begingroup$ You might want to look at this great post. The symmetry is not problem in classification as the goal of machine learning models is to make predicted distribution as close as possible to the fixed P(D), though regularizations are usually added to avoid overfitting. $\endgroup$ – doubllle Jul 19 '18 at 14:35
  • $\begingroup$ I understood the asymmetry of KL. However, I haven't understood how to use minimization of KL or Cross-Entropy differently yet. It means that when should I minimize KL and when should I minimize Cross-Entropy. I think $S_A$ is always a constant, isn't it? $\endgroup$ – Jourd Jul 19 '18 at 22:15

I suppose it is because the model usually working with the samples packed in mini-batches. For KL divergence, it can be written as $$H(q, p) = D_{KL}(p, q)+H(p) = -\sum_i{p_ilog(q_i)}$$ From the equation, we could see that KL divergence can depart into a Cross-Entropy of p and q (the first part), and a global entropy of ground truth p (the second part).

In may machine learning projects, minibatich is involved to expedite training, where the $p'$ of a minibatch may be different from the global $p$. In such a case, Cross-Entropy is relatively more robust in practical when KL divergence needs a more stable H(p) to finish her job.

  • 1
    $\begingroup$ This is being automatically flagged as low quality, probably because it is so short. At present it is more of a comment than an answer by our standards. Can you expand on it? We can also turn it into a comment. $\endgroup$ – gung May 20 at 17:49
  • 1
    $\begingroup$ The answer has been updated. $\endgroup$ – zewen liu May 20 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.