# How to write a Kullback–Leibler divergence in mathematically correct notation?

Let we have random variables $$X$$, $$Y$$, $$Z$$ and a function $$q: Z \to \mathbb{R}$$. How to write a Kullback–Leibler divergence between $$q(Z)$$ and $$p(Z|X)$$ mathematically correct?

• If we write $$D_{\text{KL}}(q || p)$$, this is ambigous. What $$p$$ do we mean: $$p(Z)$$, $$p(Z|X)$$, or $$p(Z|Y)$$?

• If we write $$D_{\text{KL}}(q(Z) || p(Z|X))$$, this seems incorrect. It is written as if Z takes on a specific value, and a result $$q(Z) \in \mathbb{R}$$ is substituted into the formula, but it's not. Kullback–Leibler divergence is meashured between distributions, not between probabilities.

Second variant seems better. But is mathematical correctness achievable?

• when im just writing it for myself, I like to do $\mathrm{KL}(q_{Z|X}||p_{Z|X})$ or as appropriate in order to distinguish marginal from conditional etc. measures, the same way people will sometimes write something like $f_x(x)$ Commented Feb 18, 2023 at 7:57

It seems that your question stems from Section 9.4 in Pattern Recognition and Machine Learning by C. Bishop, where the author used upper-case letters $$X, Z$$ to denote both random variables and arguments of likelihood functions. For the latter, more standard notations should be lower-case letters. This ambiguity may cause confusions.
Some descriptions in your post can be made more accurate as well: $$q$$ is not "a function from $$Z$$ to $$\mathbb{R}$$", instead, it is the density (here, "density" is a generic term which may stand for either the pmf when $$Z$$ is discrete or the pdf when $$Z$$ is continuous) of the latent random variable $$Z$$, which is a function from $$\mathbb{R}$$ to $$\mathbb{R}$$. To stress that $$q$$ is the density associated with $$Z$$, a subscript "$$Z$$" can be attached to $$q$$, i.e., using $$q_Z$$ to denote the density of $$Z$$. When we need to indicate the argument of $$q_Z$$ (e.g., the value of $$q_Z$$ evaluated at $$z$$), it can be made more specific as $$q_Z(z)$$ (I personally do not recommend writing $$q(Z)$$ or $$q_Z(Z)$$, which would be easily understood as a new random variable but not a distribution function). By the same token, the precise meaning of the notation "$$p(Z|X)$$" is the conditional density function of $$Z$$ given $$X = x$$ (where $$x$$ is the observed data values). When the joint density $$f$$ of $$(X, Z)$$ exists, the conditional density function of $$Z$$ given $$X = x$$, denoted by $$p_{Z|X}(z|X = x)$$ for the greatest clarity, is given by \begin{align} p_{Z|X}(z|X = x) = \frac{f(x, z)}{p_X(x)}, \end{align}
where $$p_X$$ denotes the marginal density of $$X$$. It is easy to verify that $$p_{Z|X}(z|X = x)$$ is a valid density. Oftentimes, without losing the clarity, $$p_{Z|X}(z|X = x)$$ can be abbreviated as $$p_{Z|X}(z|x)$$, or when the arguments are not needed, as $$p_{Z|X}$$ (with this notation, remember that this is a function of $$z$$, not $$x$$).
Now that both $$q_Z$$ and $$p_{Z|X}$$ are probability densities, the KL divergence between them can be unambiguously denoted as $$\boxed{D_{\text{KL}}(q_Z||p_{Z|X})}$$, whose precise expansion is: \begin{align} D_{\text{KL}}(q_Z||p_{Z|X}) = \int_\mathbb{R} q_Z(z)\log\left(\frac{q_Z(z)}{p_{Z|X}(z|x)}\right)dz. \tag{1} \end{align} Note that as $$x$$ changes, $$(1)$$ is a function of $$x$$.