# Can we use Kullback-Leibler in either direction as a loss function for probabilistic classifiers?

Suppose we are learning a probabilistic classifier $$q(x)$$ approximating a true distribution $$p(x)$$.

One natural similarity measure between distributions $$p$$ and $$q$$ is the Kullback-Leibler distance $$D_{KL}(p||q) = H(p,q) - H(p),$$ where $$H(p,q) = - \sum_x \log q(x) p(x)$$ is the cross-entropy between $$p$$ and $$q$$, and $$H(p)$$ is $$p$$'s entropy $$-\sum_x \log p(x) p(x)$$.

So we would like to find $$q$$ minimizing $$D_{KL}(p||q)$$. Because $$H(p)$$ is constant in $$q$$, this is equivalent to minimizing $$H(p,q)$$, which is the typical loss function used in, for example, neural networks.

My question is this: both $$D_{KL}(p||q)$$ and $$D_{KL}(q||p)$$, while not equal, are considered similarity measures for $$p$$ and $$q$$. So in principle we could use $$D_{KL}(q||p)$$ as the loss function of a neural network. In this case we could not replace it by $$H(q,p)$$ because both terms of $$D_{KL}(q||p)$$ do depend on $$q$$, but we could simply minimize $$D_{KL}(q||p)$$ directly.

Would that work at all? Is there any particular reason this is not done, or is it just a matter of convention and possibly convenience since $$D_{KL}(p||q)$$ can be replaced by $$H(p,q)$$ while $$D_{KL}(q||p)$$ cannot be replaced by $$H(q,p)$$? Using $$H(p,q)$$ is a nice convenience but it does not seem like a deal-breaker.

PS: this may seem like a duplicate question from "What happens if I flip targets and predictions in cross-entropy?", but it is more like an elaboration. The excellent answer in there explains why $$D_{KL}(p||q)$$ and $$H(p,q)$$ are more intuitive since $$p$$ is to be considered the "true distribution", but I do not believe that it explains why $$D_{KL}(q||p)$$ would not work since it is also a similarity measure of $$p$$ and $$q$$.