I have lots of experience computing KL divergences for straightforward discrete distributions where I have access to complete probability tables, etc. But I'm a little concerned about my current problem. I have a 2-D distribution, a Gaussian mixture, which I have a contour plot of here:

enter image description here

The above thus represents $p(\theta \mid x_1,\ldots,x_n)$ where $\theta = (\theta_1,\theta_2)$ and the $x_i$s were drawn from a Gaussian mixture model. The modes are at $(0,1)$ and $(1,-1)$.

I have two scatter plots that show MCMC sampling to estimate that posterior distribution.

enter image description here

Both of these attempt to estimate the posterior. However, I would like to have a principled way of telling which distribution is "closer" to the contour plot (the "true" distribution), hence I need the KL divergence.

I think the way I would do this is as follows. For both scatter plots, I would discretize the two dimensions, perhaps at the granularity of 0.05 for both dimensions, and compute the empirical estimate of $p(\theta \mid x_1, \ldots, x_n)$ for each of the discretized $\theta$ values. I think I would round each point in the scatter plot to its closest discretization (e.g., $(0.06,0.09)$ would turn into $(0.05, 0.10)$), and that counts as one spot for that element. Then at the end, after counting up the number of dots in each discretized bin, I'd add perhaps a count of 0.5 to each bin to smooth the distribution and then normalize. Does this make sense?

Oh, I would also have to do this same discretization for that distribution that generated the contour plot, I guess, but that's not so bad because I have the exact function so I can just call it with the bin values.

This is in python so any suggested libraries/methods that could assist in this process would be great.

Remark: I don't think this problem depends on some of the details I said earlier (especially about how $p(\theta \mid x_1, \ldots, x_n)$ gets computed) but I figured I would mention it in case.


1 Answer 1


Sorry about answering my own question, but since no one else responded, and I think I managed to resolve it, I'll answer: the discretization process I outlined in my question seems to work.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.