2
$\begingroup$

Let we have random variables $X$, $Y$, $Z$ and a function $q: Z \to \mathbb{R}$. How to write a Kullback–Leibler divergence between $q(Z)$ and $p(Z|X)$ mathematically correct?

  • If we write $D_{\text{KL}}(q || p)$, this is ambigous. What $p$ do we mean: $p(Z)$, $p(Z|X)$, or $p(Z|Y)$?

  • If we write $D_{\text{KL}}(q(Z) || p(Z|X))$, this seems incorrect. It is written as if Z takes on a specific value, and a result $q(Z) \in \mathbb{R}$ is substituted into the formula, but it's not. Kullback–Leibler divergence is meashured between distributions, not between probabilities.

Second variant seems better. But is mathematical correctness achievable?

$\endgroup$
1
  • 2
    $\begingroup$ when im just writing it for myself, I like to do $\mathrm{KL}(q_{Z|X}||p_{Z|X})$ or as appropriate in order to distinguish marginal from conditional etc. measures, the same way people will sometimes write something like $f_x(x)$ $\endgroup$ Commented Feb 18, 2023 at 7:57

1 Answer 1

2
$\begingroup$

It seems that your question stems from Section 9.4 in Pattern Recognition and Machine Learning by C. Bishop, where the author used upper-case letters $X, Z$ to denote both random variables and arguments of likelihood functions. For the latter, more standard notations should be lower-case letters. This ambiguity may cause confusions.

Some descriptions in your post can be made more accurate as well: $q$ is not "a function from $Z$ to $\mathbb{R}$", instead, it is the density (here, "density" is a generic term which may stand for either the pmf when $Z$ is discrete or the pdf when $Z$ is continuous) of the latent random variable $Z$, which is a function from $\mathbb{R}$ to $\mathbb{R}$. To stress that $q$ is the density associated with $Z$, a subscript "$Z$" can be attached to $q$, i.e., using $q_Z$ to denote the density of $Z$. When we need to indicate the argument of $q_Z$ (e.g., the value of $q_Z$ evaluated at $z$), it can be made more specific as $q_Z(z)$ (I personally do not recommend writing $q(Z)$ or $q_Z(Z)$, which would be easily understood as a new random variable but not a distribution function). By the same token, the precise meaning of the notation "$p(Z|X)$" is the conditional density function of $Z$ given $X = x$ (where $x$ is the observed data values). When the joint density $f$ of $(X, Z)$ exists, the conditional density function of $Z$ given $X = x$, denoted by $p_{Z|X}(z|X = x)$ for the greatest clarity, is given by \begin{align} p_{Z|X}(z|X = x) = \frac{f(x, z)}{p_X(x)}, \end{align}
where $p_X$ denotes the marginal density of $X$. It is easy to verify that $p_{Z|X}(z|X = x)$ is a valid density. Oftentimes, without losing the clarity, $p_{Z|X}(z|X = x)$ can be abbreviated as $p_{Z|X}(z|x)$, or when the arguments are not needed, as $p_{Z|X}$ (with this notation, remember that this is a function of $z$, not $x$).

Now that both $q_Z$ and $p_{Z|X}$ are probability densities, the KL divergence between them can be unambiguously denoted as $\boxed{D_{\text{KL}}(q_Z||p_{Z|X})}$, whose precise expansion is: \begin{align} D_{\text{KL}}(q_Z||p_{Z|X}) = \int_\mathbb{R} q_Z(z)\log\left(\frac{q_Z(z)}{p_{Z|X}(z|x)}\right)dz. \tag{1} \end{align} Note that as $x$ changes, $(1)$ is a function of $x$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.