KL divergence for joint probability distributions? I have a pair of joint probability distributions. I want to measure their similarity/dissimilarity. If they were single-dimensional probability distributions, then I could measure the Kullback–Leibler (KL) divergence or the Jensen–Shannon divergence between them. What can I do in the case of a pair of joint probability distributions?
If it is useful, one can assume that I have two tables, each containing a large number of samples (as many as desired) from each of the joint probability distributions.
 A: KL divergence is defined between two distributions, period. If this is marginal or joint distributions is immaterial. You want them to have the same support. So you do the same as in single dimension. Asker in a comment says

Thanks for your useful answer. In the KL divergence, we must calculate
$\log p/q$ for probability distributions $p$ and $q$; how can this
calculation be done for joint distributions? (Sorry if this is a basic
question!)

$p$, $q$ are density functions, so they have values that are non-negative real numbers. So the quotient $p/q$ (assuming it is defined where needed, which it will be when the support of $p$ is included in the support of $q$) is a non-negative real number.
That conclusion does not at all depend on how many arguments the density functions $p, q$ have (they will normally have the same number of arguments). So the calculation of KL divergence is in principle the same for marginal and joint distributions, although the multivariate case might in practice be more involved.
For some examples see

*

*KL divergence between two multivariate Gaussians


*Kullback-Leibler divergence between multivariate t and the multivariate normal?


*Efficiently computing pairwise KL divergence between multiple diagonal-covariance Gaussian distributions
