4
$\begingroup$

If you draw samples of unknown distribution, how can you measure the KL-divergence between the unknown distribution and a gaussian distribution with zero mean and unit variance N(0,1)?

Can we use the moments measured from the drawn samples? Or how can we approach this?

Additionally, it would be interesting if the method is differentiable.

Some related work I found:

In [1], the methods are implemented to compare unknown distributions and you only have the samples from both distributions.

In [2], the authors state, "It has been known for some time that in the case of the Gaussian distribution, matching the first two moments of the original density yields the optimal approximation in terms of minimizing the KLD." and they prove that also true for some hyperspherical probability distributions, namely the von Mises–Fisher and the Watson distribution.

References:
[1] KL-Divergence Estimators, https://github.com/nhartland/KL-divergence-estimators
[2] Kurz, Gerhard, Florian Pfaff, and Uwe D. Hanebeck. "Kullback–Leibler Divergence and Moment Matching for Hyperspherical Probability Distributions", https://www.researchgate.net/publication/305449272_Kullback-Leibler_Divergence_and_Moment_Matching_for_Hyperspherical_Probability_Distributions

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ The KL divergence is a measure that can be computed if you have two distribution functions. If you have a sample, then you need to apply some way to estimate the distribution. This can be done in different ways, and how you do it depends on what you know about the distribution. $\endgroup$
    – Sextus Empiricus
    Mar 30 at 14:28
  • $\begingroup$ Related: stats.stackexchange.com/questions/610174 $\endgroup$
    – Sextus Empiricus
    Mar 30 at 14:28

1 Answer 1

Reset to default
0
$\begingroup$

You could do sample based KL-divergence if you have the likelihood of the unknown distribution.

Get the sample $x_n\sim p(x)$ and likelihood from the unknown distribution $p(x_n)$ and do likelihood estimation on the Gaussian distribution $q(x)$. Then, evaluate KL as follows:

$$ KL(P||Q)= \sum_{x\in\mathrm{X}}^n p(x_n) \operatorname{log}\frac{p(x_n)}{q(x_n)} $$

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ The question is how you do this. The Gaussian distribution $q(x)$ is known but how do you get $p(x)$? "if you have the likelihood of the unknown distribution". If the distribution is unknown, in what situation would one have the likelihood and how is the likelihood gonna help with computing the divergence? $\endgroup$
    – Sextus Empiricus
    Mar 30 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.