4
$\begingroup$

If you draw samples of unknown distribution, how can you measure the KL-divergence between the unknown distribution and a gaussian distribution with zero mean and unit variance N(0,1)?

Can we use the moments measured from the drawn samples? Or how can we approach this?

Additionally, it would be interesting if the method is differentiable.

Some related work I found:

In [1], the methods are implemented to compare unknown distributions and you only have the samples from both distributions.

In [2], the authors state, "It has been known for some time that in the case of the Gaussian distribution, matching the first two moments of the original density yields the optimal approximation in terms of minimizing the KLD." and they prove that also true for some hyperspherical probability distributions, namely the von Mises–Fisher and the Watson distribution.

References:
[1] KL-Divergence Estimators, https://github.com/nhartland/KL-divergence-estimators
[2] Kurz, Gerhard, Florian Pfaff, and Uwe D. Hanebeck. "Kullback–Leibler Divergence and Moment Matching for Hyperspherical Probability Distributions", https://www.researchgate.net/publication/305449272_Kullback-Leibler_Divergence_and_Moment_Matching_for_Hyperspherical_Probability_Distributions

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ The KL divergence is a measure that can be computed if you have two distribution functions. If you have a sample, then you need to apply some way to estimate the distribution. This can be done in different ways, and how you do it depends on what you know about the distribution. $\endgroup$
    – Sextus Empiricus
    Mar 30 at 14:28
  • $\begingroup$ Related: stats.stackexchange.com/questions/610174 $\endgroup$
    – Sextus Empiricus
    Mar 30 at 14:28

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.