Each distribution is represented with an array of arrays with PMF values.

UPD 1: I have $P=(p_1, ... , p_n)$ where $P$ is a distribution of distributions and $p_i=(p_i^1, ..., p_i^m)$. My task is to compute $D_{KL}(P, Q)$.

UPD 2: Each $p_i$ is PMF and $\sum_j p_i^j=1$ for each i.


closed as unclear what you're asking by Michael Chernick, Carl, gung, hxd1011, Juho Kokkala Apr 24 '17 at 16:50

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ stats.stackexchange.com/questions/211175/… $\endgroup$ – kjetil b halvorsen Apr 21 '17 at 17:41
  • 1
    $\begingroup$ Is each $p_i$ a PMF? i.e. $\sum_jp_i^j=1$? Or is it only $P$ that is a PMF? i.e. $\sum_i\sum_jp_i^j=1$? Note that KL divergence is an expectation, so it only makes sense on "entire" PMF's (i.e. sum is 1). If you already have PMFs (vs. PDFs) then you can just sum bin-probabilities (i.e. the multi-dimensional part would only comes in to convert from density to mass, via bin volume). $\endgroup$ – GeoMatt22 Apr 21 '17 at 22:52
  • $\begingroup$ @GeoMatt22 Yes, each $p_i$ is PMF and $\sum_j p_i^j=1$ for each i. $\endgroup$ – Anton Karazeev Apr 22 '17 at 4:13
  • $\begingroup$ In what sense are the distributions "multidimensional"? What is $Q$? $\endgroup$ – Juho Kokkala Apr 24 '17 at 16:50
  • $\begingroup$ @JuhoKokkala $Q$ is like $P$, but $Q$. See the UPD 1. $\endgroup$ – Anton Karazeev Apr 24 '17 at 18:48

The KL-divergence does not depend on the dimensionality of the distribution - since a pmf must always be one-dimensional. (ie, what would it mean if $P(X = k)$ was a vector?)

What I mean is, the integral/summation in KL-divergence is with respect to $\mathbf{x}$, not $\theta$. For two distributions $p(\mathbf{x})$ and $q(\mathbf{x})$, you can write:

$$D_{KL}(p|q) = \int_\mathcal{X} p(\mathbf{x})\log\frac{p(\mathbf{x})}{q(\mathbf{x})}d\mathbf{x}$$

  • 2
    $\begingroup$ He is talking about KL between multivariate distributions. The question is legitimate. Your answer on the other hand is not. $\endgroup$ – Cowboy Trader Apr 21 '17 at 19:01
  • 1
    $\begingroup$ I don't see any issue with his answer. The formula he referenced uses multivariate distributions. We know this because there are bold letters in them. $\endgroup$ – jjet Apr 21 '17 at 19:14
  • $\begingroup$ Yes, a multivariate distribution would imply the variable of interest, $\mathbf{x}$, is a vector. Althought I would appreciate OP clarifying his question a bit, if I'm misinterpreting. $\endgroup$ – Tim Atreides Apr 21 '17 at 19:15
  • $\begingroup$ @TimAtreides actually distribution doesn't have to "be always one-dimensional". What if I have some machine learning task where value of each feature for given object has its own distribution as this value is a random variable? How should I compute KL-divergence between two such objects? $\endgroup$ – Anton Karazeev Apr 21 '17 at 20:03
  • 1
    $\begingroup$ Oh, dude! Your problem is much easier, then. Just let $\mathbf{p} = (p_1,...,p_n)$ and $\mathbf{q} = (q_1,...,q_n)$ and compute the following: $$D_{KL} = \sum_i p_i \log\frac{p_i}{q_i}$$ Which I believe is the correct way to compute the empirical KL divergence. $\endgroup$ – Tim Atreides Apr 21 '17 at 20:47

Not the answer you're looking for? Browse other questions tagged or ask your own question.