# KL divergence for joint probability distributions?

I have a pair of joint probability distributions. I want to measure their similarity/dissimilarity. If they were single-dimensional probability distributions, then I could measure the Kullback–Leibler (KL) divergence or the Jensen–Shannon divergence between them. What can I do in the case of a pair of joint probability distributions?

If it is useful, one can assume that I have two tables, each containing a large number of samples (as many as desired) from each of the joint probability distributions.

• @kjetilbhalvorsen Thanks for your useful answer. In the KL divergence, we must calculate $\log p/q$ for probability distributions $p$ and $q$; how can this calculation be done for joint distributions? (Sorry if this is a basic question!) – rhombidodecahedron Mar 22 at 13:44
• Are you trying to compute the KLD between two joint probability density functions $q$ and $p$ with the same support? Then you need to simply calculate that. See the multivariate normal case (very popular because of VAEs). – Firebug Mar 22 at 14:34

## 1 Answer

KL divergence is defined between two distributions, period. If this is marginal or joint distributions is immaterial. You want them to have the same support. So you do the same as in single dimension. Asker in a comment says

Thanks for your useful answer. In the KL divergence, we must calculate $$\log p/q$$ for probability distributions $$p$$ and $$q$$; how can this calculation be done for joint distributions? (Sorry if this is a basic question!)

$$p$$, $$q$$ are density functions, so they have values that are non-negative real numbers. So the quotient $$p/q$$ (assuming it is defined where needed, which it will be when the support of $$p$$ is included in the support of $$q$$) is a non-negative real number.

That conclusion does not at all depend on how many arguments the density functions $$p, q$$ have (they will normally have the same number of arguments). So the calculation of KL divergence is in principle the same for marginal and joint distributions, although the multivariate case might in practice be more involved.

For some examples see

• Thanks, I think it makes good sense to me. Yes, my distributions have identical support. I guess I then need to just integrate over each of the arguments to $p$ and $q$? i.e. instead of $\int_x p(x) \log\left( \dfrac{p(x)}{q(x)} \right) \,\text{d}x$ I will do $\int_y \int_x p(x,y) \log\left( \dfrac{p(x,y)}{q(x,y)} \right) \,\text{d}x\,\text{d}y$? – rhombidodecahedron Mar 22 at 15:33
• Yes, so it is.. – kjetil b halvorsen Mar 22 at 15:34