isKL Divergence Asymmetry at Zero So, I'm aware that the KL divergence is asymmetric, and furthermore that it has to be asymmetric in order to be compatible with Bayes' Law. But there's one particular asymmetry that confuses me.
Let $$KL(P\Vert Q) = \sum_x P(x) \lg\frac{P(x)}{Q(x)}$$
across whatever the domain of $P,Q$ is. Then if $P(x)$ goes to zero, the equation remains well defined; the contribution to $KL(P\Vert Q)$ tends smoothly to zero as well. But if $Q(x)$ goes to zero, the integrand diverges, and $KL(P\Vert Q)$ can explode.
From the Bayesian perspective, though, these two cases should be symmetric. Certainly, if you start with an infinite certainty that something cannot occur ($Q(x)=0$), no finite amount of evidence will ever convince you otherwise ($KL(P\Vert Q)=\infty$ makes sense). But if you start with finite certainty, it should also take you an infinite amount of evidence to convince you that something is completely impossible ($P(x)=0$, but $KL(P\Vert Q)\neq\infty)$.
What am I missing here?
 A: The last part of your statement is incorrect. Suppose for example that $x$ is the average number of rainy days in a year in some location that you know nothing about, so a-priori $x = 0$ is possible with some (possibly) finite probability. But it only takes observing a single rainy day to convince you that $x=0$ is completely impossible.
A: I am not sure what you mean when stating that the KL divergence "has to be asymmetric in order to be compatible with Bayes' Law": Does it have to be compatible with Bayes's Law? And, if so, in what sense? Still, I don't think that this is crucial to your question (and if it is, please be more specific).
There are many ways to understand the KL divergence. From a Bayesian perspective, you can think of $Q(x)$ as a prior distribution and $P(x)$ as a posterior distribution in the sense that the data was used to estimate it. $P(x)$ is not necessarily the posterior update of $Q(x)$, but it could be. Note that even though $Q(x) = 0$ signifies certainty that $x$ will not occur, $Q(x) > 0$ does not mean certainty that $x$ will occur. $Q(x) > 0$ signifies a positive probability that $x$ can occur.
As an example, let $Q(x)$ be a mixture of $K$ parametric distributions,
$$Q(x) = \sum_{k = 1} ^K w_k Q_k(x|\theta_k),$$
where $w_k>0$ are the weights and $\sum w_k = 1$. Let the posterior distribution be
$$P(x) = \sum_{k = 1} ^K \hat{w}_k P_k(x|\hat{\theta}_k),\,\, \hat{w}_k \geq 0, \sum \hat{w}_k = 1.$$
For example if a sample is drawn from a subset of $K$ populations, but you do not know beforehand which, then some $\hat{w}_k = 0$.
In this scenario, if for some $x$, $Q(x) = 0$ and $P(x) > 0$, then $D_{KL} \rightarrow \infty$ seems reasonable. The difference between $P(x)$ and $Q(x)$ tends to infinity, and leads us to the conclusion that $P(x)$ has been sampled outside of the $K$ populations. But if for some $x$, $Q(x) > 0$ and $P(x) = 0$, this simply means that some $\hat{w}_k = 0$, and is in accordance with the overall model: samples are taken from a subset of the $K$ candidate populations.
