# How do I calculate KL-divergence between two multidimensional distributions? [closed]

Each distribution is represented with an array of arrays with PMF values.

UPD 1: I have $P=(p_1, ... , p_n)$ where $P$ is a distribution of distributions and $p_i=(p_i^1, ..., p_i^m)$. My task is to compute $D_{KL}(P, Q)$.

UPD 2: Each $p_i$ is PMF and $\sum_j p_i^j=1$ for each i.

## closed as unclear what you're asking by Michael Chernick, Carl, gung♦, hxd1011, Juho KokkalaApr 24 '17 at 16:50

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

• stats.stackexchange.com/questions/211175/… – kjetil b halvorsen Apr 21 '17 at 17:41
• Is each $p_i$ a PMF? i.e. $\sum_jp_i^j=1$? Or is it only $P$ that is a PMF? i.e. $\sum_i\sum_jp_i^j=1$? Note that KL divergence is an expectation, so it only makes sense on "entire" PMF's (i.e. sum is 1). If you already have PMFs (vs. PDFs) then you can just sum bin-probabilities (i.e. the multi-dimensional part would only comes in to convert from density to mass, via bin volume). – GeoMatt22 Apr 21 '17 at 22:52
• @GeoMatt22 Yes, each $p_i$ is PMF and $\sum_j p_i^j=1$ for each i. – Anton Karazeev Apr 22 '17 at 4:13
• In what sense are the distributions "multidimensional"? What is $Q$? – Juho Kokkala Apr 24 '17 at 16:50
• @JuhoKokkala $Q$ is like $P$, but $Q$. See the UPD 1. – Anton Karazeev Apr 24 '17 at 18:48

The KL-divergence does not depend on the dimensionality of the distribution - since a pmf must always be one-dimensional. (ie, what would it mean if $P(X = k)$ was a vector?)
What I mean is, the integral/summation in KL-divergence is with respect to $\mathbf{x}$, not $\theta$. For two distributions $p(\mathbf{x})$ and $q(\mathbf{x})$, you can write:
$$D_{KL}(p|q) = \int_\mathcal{X} p(\mathbf{x})\log\frac{p(\mathbf{x})}{q(\mathbf{x})}d\mathbf{x}$$
• Yes, a multivariate distribution would imply the variable of interest, $\mathbf{x}$, is a vector. Althought I would appreciate OP clarifying his question a bit, if I'm misinterpreting. – Tim Atreides Apr 21 '17 at 19:15
• Oh, dude! Your problem is much easier, then. Just let $\mathbf{p} = (p_1,...,p_n)$ and $\mathbf{q} = (q_1,...,q_n)$ and compute the following: $$D_{KL} = \sum_i p_i \log\frac{p_i}{q_i}$$ Which I believe is the correct way to compute the empirical KL divergence. – Tim Atreides Apr 21 '17 at 20:47