7
$\begingroup$

I have a pair of joint probability distributions. I want to measure their similarity/dissimilarity. If they were single-dimensional probability distributions, then I could measure the Kullback–Leibler (KL) divergence or the Jensen–Shannon divergence between them. What can I do in the case of a pair of joint probability distributions?

If it is useful, one can assume that I have two tables, each containing a large number of samples (as many as desired) from each of the joint probability distributions.

Improve this question
$\endgroup$
2
  • $\begingroup$ @kjetilbhalvorsen Thanks for your useful answer. In the KL divergence, we must calculate $\log p/q$ for probability distributions $p$ and $q$; how can this calculation be done for joint distributions? (Sorry if this is a basic question!) $\endgroup$
    – rhombidodecahedron
    Commented Mar 22, 2021 at 13:44
  • 2
    $\begingroup$ Are you trying to compute the KLD between two joint probability density functions $q$ and $p$ with the same support? Then you need to simply calculate that. See the multivariate normal case (very popular because of VAEs). $\endgroup$
    – Firebug
    Commented Mar 22, 2021 at 14:34

1 Answer 1

Reset to default
8
$\begingroup$

KL divergence is defined between two distributions, period. If this is marginal or joint distributions is immaterial. You want them to have the same support. So you do the same as in single dimension. Asker in a comment says

Thanks for your useful answer. In the KL divergence, we must calculate $\log p/q$ for probability distributions $p$ and $q$; how can this calculation be done for joint distributions? (Sorry if this is a basic question!)

$p$, $q$ are density functions, so they have values that are non-negative real numbers. So the quotient $p/q$ (assuming it is defined where needed, which it will be when the support of $p$ is included in the support of $q$) is a non-negative real number.

That conclusion does not at all depend on how many arguments the density functions $p, q$ have (they will normally have the same number of arguments). So the calculation of KL divergence is in principle the same for marginal and joint distributions, although the multivariate case might in practice be more involved.

For some examples see

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thanks, I think it makes good sense to me. Yes, my distributions have identical support. I guess I then need to just integrate over each of the arguments to $p$ and $q$? i.e. instead of $\int_x p(x) \log\left( \dfrac{p(x)}{q(x)} \right) \,\text{d}x$ I will do $\int_y \int_x p(x,y) \log\left( \dfrac{p(x,y)}{q(x,y)} \right) \,\text{d}x\,\text{d}y$? $\endgroup$
    – rhombidodecahedron
    Commented Mar 22, 2021 at 15:33
  • 1
    $\begingroup$ Yes, so it is.. $\endgroup$
    – kjetil b halvorsen ♦
    Commented Mar 22, 2021 at 15:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.