Is it possible to compute the Jensen-Shannon divergence between a discrete and a continuous probability distribution, e.g. between a standard normal distribution and a distribution taking the values 1,2,3 each with probability 1/3? Or are there any other divergences which can be used as a similarity measure of discrete and continuous distributions.

  • $\begingroup$ related question: stats.stackexchange.com/questions/69125/… $\endgroup$ – Pat Sep 4 '13 at 11:11
  • $\begingroup$ I have to ask why you'd want to measure the divergence a continuous and discrete distribution. It doesn't feel like a very natural thing to do at all. If you give us some more details as to what you're wanting to apply it to, people may be able to steer you towards a solution. $\endgroup$ – Pat Sep 4 '13 at 11:12
  • $\begingroup$ @Pat: This is not uncommon in dynamic state estimation, e.g. the Unscented Kalman Filter (UKF) approximates (using Moment Matching) a d-dimensional Normal distribution by a discrete probability distribution (consisting of 2*d+1 components) with same mean and covariance. Sometimes, the quality can be improved by approximating the original distribution with a more complex discrete distribution. I am interested in a good distance Measure which can be used to perform this kind of optimization. $\endgroup$ – Igor Sep 4 '13 at 11:36
  • $\begingroup$ Okay. I'm not familiar with the UKF. From what I can read online you take a set of points with the same mean and covariance as your normal distribution, pass them through the system's non-linear transform, and use the transformed points to re-estimate the normal distribution in the new space by moment matching? In that case wouldn't it be best to measure the divergence of the re-estimated normal from the 'ideal' normal (probably what you would get if you used an infinite number of sigma points)? Then you're comparing two continuous distributions to one another, and avoid the problem entirely. $\endgroup$ – Pat Sep 4 '13 at 12:03
  • $\begingroup$ Yes, this would be the best strategy. Unfortunately, this approach is often computationally burdensome (this obviously depends on the system function), which motivates the use of approximations of continuous probability distributions by discrete distributions. $\endgroup$ – Igor Sep 4 '13 at 12:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.