Say $X$ is a discrete random variable with cardinality $|X|$ and $Y$ is a discrete random variable with cardinality $|Y|$.

Does it make sense to talk about the KL divergences $D_{KL}(X||Y)$ or $D_{KL}(Y||X)$ of these 2 probability distributions if $|X| \neq |Y|$ ? If so, how does one compute it ?

  • 7
    $\begingroup$ From the definition: "The K-L divergence is only defined if $P$ and $Q$ both sum to $1$ and if $Q(i)=0$ implies $P(i)=0$ for all $i$ (absolute continuity)." Therefore the answer in general is: no. $\endgroup$
    – user10525
    Oct 15, 2012 at 9:26
  • 4
    $\begingroup$ A related question. Didier's answer there, in particular, is highly relevant. $\endgroup$
    – cardinal
    Oct 15, 2012 at 10:07
  • 1
    $\begingroup$ An example: If $X$ has the possible values $0,1,2$ and $Y$ has the possible values $0,1,2,3$ and your observation is $3$, then $X$ is excluded as a model by usual, determinisic logic! Sothe power to discriminate between these twomodels is infinitely large, and you could , if you want, define KL-divergence from $X$ to$Y$ to be $\infty$, if that makes sense. The KL-divergence in the other direction could be finite. $\endgroup$ Oct 15, 2012 at 14:46
  • 1
    $\begingroup$ Yes, it makes sense to think of $D_{KL}(Y||X) = \inf$ and compute $D_{KL}(X||Y)$ for values $0,1,2$ in your example. $\endgroup$
    – locke14
    Oct 17, 2012 at 11:19


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.