We know that for discrete variables \begin{equation} D(p(x),q(x))=\mathbb{E}_{p}\left(\log\frac{p(x)}{q(x)}\right) \end{equation} where $p(x)$ and $q(x)$ are probability mass functions. Can this be extended to \begin{equation} D(p(x,y,z),q(x,y,z))=\mathbb{E}_{p}\left(\log\frac{p(x,y,z)}{q(x,y,z)}\right) \end{equation} ? where now $p(x,y,z)$ and $q(x,y,z)$ are joint probability mass functions. Is the notation correct?
$\begingroup$
$\endgroup$
3
-
3$\begingroup$ Yes, that's correct. The way you've written KL divergence using expectations is valid for any pair of distributions defined on the same space (not limited to discrete values; also works for reals, vectors, etc.). Notation looks fine to me, as long as you specify that $D(\cdot, \cdot)$ means KL divergence. Everyone writes things a little bit differently. $\endgroup$– user20160Commented Nov 27, 2019 at 8:26
-
$\begingroup$ Yes, and see here for an explanation and interpretation of that expectation. $\endgroup$– kjetil b halvorsen ♦Commented Nov 27, 2019 at 10:57
-
$\begingroup$ @user20160: do you want to post your comment(s) as an answer? Better to have a short answer than no answer at all. Anyone who has a better answer can post it. $\endgroup$– Sycorax ♦Commented Feb 17, 2021 at 6:57
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Yes, that's correct. The way you've written KL divergence using expectations is valid for any pair of distributions defined on the same space (not limited to discrete values; also works for reals, vectors, etc.). Notation looks fine to me, as long as you specify that 𝐷(⋅,⋅) means KL divergence. Everyone writes things a little bit differently. – user20160
I've created this answer from a comment so that this question is answered. Better to have a short answer than no answer at all.
