# KL divergence for joint probability distributions?

I have a pair of joint probability distributions. I want to measure their similarity/dissimilarity. If they were single-dimensional probability distributions, then I could measure the Kullback–Leibler (KL) divergence or the Jensen–Shannon divergence between them. What can I do in the case of a pair of joint probability distributions?

If it is useful, one can assume that I have two tables, each containing a large number of samples (as many as desired) from each of the joint probability distributions.

• @kjetilbhalvorsen Thanks for your useful answer. In the KL divergence, we must calculate $\log p/q$ for probability distributions $p$ and $q$; how can this calculation be done for joint distributions? (Sorry if this is a basic question!) Mar 22, 2021 at 13:44
• Are you trying to compute the KLD between two joint probability density functions $q$ and $p$ with the same support? Then you need to simply calculate that. See the multivariate normal case (very popular because of VAEs). Mar 22, 2021 at 14:34

Thanks for your useful answer. In the KL divergence, we must calculate $$\log p/q$$ for probability distributions $$p$$ and $$q$$; how can this calculation be done for joint distributions? (Sorry if this is a basic question!)
$$p$$, $$q$$ are density functions, so they have values that are non-negative real numbers. So the quotient $$p/q$$ (assuming it is defined where needed, which it will be when the support of $$p$$ is included in the support of $$q$$) is a non-negative real number.
That conclusion does not at all depend on how many arguments the density functions $$p, q$$ have (they will normally have the same number of arguments). So the calculation of KL divergence is in principle the same for marginal and joint distributions, although the multivariate case might in practice be more involved.
• Thanks, I think it makes good sense to me. Yes, my distributions have identical support. I guess I then need to just integrate over each of the arguments to $p$ and $q$? i.e. instead of $\int_x p(x) \log\left( \dfrac{p(x)}{q(x)} \right) \,\text{d}x$ I will do $\int_y \int_x p(x,y) \log\left( \dfrac{p(x,y)}{q(x,y)} \right) \,\text{d}x\,\text{d}y$? Mar 22, 2021 at 15:33