8
$\begingroup$

Is the KL divergence not defined because uniform has bounded support and gaussian has unbounded support?

How else can I calculate the distance of my gaussian to a 'maximum entropy' distribution if I can't use the uniform distribution?

$\endgroup$
1
  • 2
    $\begingroup$ The distance is infinite. $\endgroup$
    – Xi'an
    May 21, 2019 at 6:28

2 Answers 2

8
$\begingroup$

The KL divergence $$ KL\left(P \middle\| Q\right) = \int \log \frac{d P}{d Q }dP $$ is only defined if the Radon-Nikodym derivative exists, which is when $P$ is dominated by $Q$ (written $P \ll Q$). This means that there can't be any sets $A$ where $P(A) > 0$ and $Q(A) = 0$, otherwise we would be dividing by zero.

In your case, $p$ is the density of the uniform random variable, and $q$ is the density of the normal random variable (they are both dominated by the Lebesgue measure), so you could calculate $$ KL\left(P \middle\| Q\right) = \int \log \frac{p(x)}{q(x)}p(x)dx, $$ but you couldn't calculate $KL\left(Q \middle\| P\right)$. You can calculate $KL\left(P \middle\| Q\right)$ because there are no sets $A$ such that $\int_A p(x) dx > 0$ and $\int_A q(x) dx = 0$.

$\endgroup$
2
  • $\begingroup$ Yes, unfortunately I need KL(Q||P). j $\endgroup$
    – user3180
    May 21, 2019 at 4:11
  • 1
    $\begingroup$ Technically correct but I think the question probably meant an (improper) unbounded uniform distribution, because it says "distance to a maximum entropy distribution". $\endgroup$
    – danijar
    Sep 15, 2022 at 14:34
2
$\begingroup$

How else can I calculate the distance of my gaussian to a 'maximum entropy' distribution if I can't use the uniform distribution?

You basically answered your question, you can use the entropy of the distribution as a measure of randomness. For a Gaussian distribution, it can be computed directly from the standard standard deviation (and you could also directly use the standard deviation as a measure of randomness):

$$\operatorname{H}_{p}[X] = \operatorname{E}_{p}[-\ln p(x)] = \frac{1}{2}\ln\big(2\pi\sigma^2\big)+\frac{1}{2}$$

The KL from some distribution $q$ to a uniform distribution $p$ actually contains two terms, the negative entropy of the first distribution and the cross entropy between the two distributions. Because the log probability of an unbounded uniform distribution is constant, the cross entropy is a constant:

$$ \operatorname{KL}[q(x) \,\|\, p(x)] = \operatorname{E}_{q}[\ln q(x) - \ln p(x)] \\ = \operatorname{E}_{q}[\ln q(x)] - \operatorname{E}_{q}[\ln p(x)] \\ = -\operatorname{H}[q(x)] - \operatorname{E}_{q}[\mathrm{const}] \\ = -\operatorname{H}[q(x)] - \mathrm{const} $$

For categorical variables, the constant is the log probability of uniform categorical distribution, which is $\ln \frac{1}{N}=-\ln N$. For continuous variables, you can't have an unbounded uniform distribution because it would have an infinite normalizer. A trick is to instead consider an improper uniform distribution that is unnormalized, $p(x)\propto 1$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.