Suppose we are learning a probabilistic classifier $q(x)$ approximating a true distribution $p(x)$.

One natural similarity measure between distributions $p$ and $q$ is the Kullback-Leibler distance $$D_{KL}(p||q) = H(p,q) - H(p),$$ where $H(p,q) = - \sum_x \log q(x) p(x)$ is the cross-entropy between $p$ and $q$, and $H(p)$ is $p$'s entropy $-\sum_x \log p(x) p(x)$.

So we would like to find $q$ minimizing $D_{KL}(p||q)$. Because $H(p)$ is constant in $q$, this is equivalent to minimizing $H(p,q)$, which is the typical loss function used in, for example, neural networks.

My question is this: both $D_{KL}(p||q)$ and $D_{KL}(q||p)$, while not equal, are considered similarity measures for $p$ and $q$. So in principle we could use $D_{KL}(q||p)$ as the loss function of a neural network. In this case we could not replace it by $H(q,p)$ because both terms of $D_{KL}(q||p)$ do depend on $q$, but we could simply minimize $D_{KL}(q||p)$ directly.

Would that work at all? Is there any particular reason this is not done, or is it just a matter of convention and possibly convenience since $D_{KL}(p||q)$ can be replaced by $H(p,q)$ while $D_{KL}(q||p)$ cannot be replaced by $H(q,p)$? Using $H(p,q)$ is a nice convenience but it does not seem like a deal-breaker.

PS: this may seem like a duplicate question from "What happens if I flip targets and predictions in cross-entropy?", but it is more like an elaboration. The excellent answer in there explains why $D_{KL}(p||q)$ and $H(p,q)$ are more intuitive since $p$ is to be considered the "true distribution", but I do not believe that it explains why $D_{KL}(q||p)$ would not work since it is also a similarity measure of $p$ and $q$.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.