# KL divergence between 2 distributions with unequal cardinalities?

Say $X$ is a discrete random variable with cardinality $|X|$ and $Y$ is a discrete random variable with cardinality $|Y|$.

Does it make sense to talk about the KL divergences $D_{KL}(X||Y)$ or $D_{KL}(Y||X)$ of these 2 probability distributions if $|X| \neq |Y|$ ? If so, how does one compute it ?

• From the definition: "The K-L divergence is only defined if $P$ and $Q$ both sum to $1$ and if $Q(i)=0$ implies $P(i)=0$ for all $i$ (absolute continuity)." Therefore the answer in general is: no. – user10525 Oct 15 '12 at 9:26
• A related question. Didier's answer there, in particular, is highly relevant. – cardinal Oct 15 '12 at 10:07
• An example: If $X$ has the possible values $0,1,2$ and $Y$ has the possible values $0,1,2,3$ and your observation is $3$, then $X$ is excluded as a model by usual, determinisic logic! Sothe power to discriminate between these twomodels is infinitely large, and you could , if you want, define KL-divergence from $X$ to$Y$ to be $\infty$, if that makes sense. The KL-divergence in the other direction could be finite. – kjetil b halvorsen Oct 15 '12 at 14:46
• Yes, it makes sense to think of $D_{KL}(Y||X) = \inf$ and compute $D_{KL}(X||Y)$ for values $0,1,2$ in your example. – locke14 Oct 17 '12 at 11:19