3
$\begingroup$

Say $X$ is a discrete random variable with cardinality $|X|$ and $Y$ is a discrete random variable with cardinality $|Y|$.

Does it make sense to talk about the KL divergences $D_{KL}(X||Y)$ or $D_{KL}(Y||X)$ of these 2 probability distributions if $|X| \neq |Y|$ ? If so, how does one compute it ?

Improve this question
$\endgroup$
4
  • 7
    $\begingroup$ From the definition: "The K-L divergence is only defined if $P$ and $Q$ both sum to $1$ and if $Q(i)=0$ implies $P(i)=0$ for all $i$ (absolute continuity)." Therefore the answer in general is: no. $\endgroup$
    – user10525
    Oct 15, 2012 at 9:26
  • 4
    $\begingroup$ A related question. Didier's answer there, in particular, is highly relevant. $\endgroup$
    – cardinal
    Oct 15, 2012 at 10:07
  • 1
    $\begingroup$ An example: If $X$ has the possible values $0,1,2$ and $Y$ has the possible values $0,1,2,3$ and your observation is $3$, then $X$ is excluded as a model by usual, determinisic logic! Sothe power to discriminate between these twomodels is infinitely large, and you could , if you want, define KL-divergence from $X$ to$Y$ to be $\infty$, if that makes sense. The KL-divergence in the other direction could be finite. $\endgroup$
    – kjetil b halvorsen ♦
    Oct 15, 2012 at 14:46
  • 1
    $\begingroup$ Yes, it makes sense to think of $D_{KL}(Y||X) = \inf$ and compute $D_{KL}(X||Y)$ for values $0,1,2$ in your example. $\endgroup$
    – locke14
    Oct 17, 2012 at 11:19

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.