KL divergence between gaussian and uniform distribution

Question

Is the KL divergence not defined because uniform has bounded support and gaussian has unbounded support?

How else can I calculate the distance of my gaussian to a 'maximum entropy' distribution if I can't use the uniform distribution?

Answer 1

The KL divergence $$ KL\left(P \middle\| Q\right) = \int \log \frac{d P}{d Q }dP $$ is only defined if the Radon-Nikodym derivative exists, which is when $P$ is dominated by $Q$ (written $P \ll Q$). This means that there can't be any sets $A$ where $P(A) > 0$ and $Q(A) = 0$, otherwise we would be dividing by zero.

In your case, $p$ is the density of the uniform random variable, and $q$ is the density of the normal random variable (they are both dominated by the Lebesgue measure), so you could calculate $$ KL\left(P \middle\| Q\right) = \int \log \frac{p(x)}{q(x)}p(x)dx, $$ but you couldn't calculate $KL\left(Q \middle\| P\right)$. You can calculate $KL\left(P \middle\| Q\right)$ because there are no sets $A$ such that $\int_A p(x) dx > 0$ and $\int_A q(x) dx = 0$.

Answer 2

How else can I calculate the distance of my gaussian to a 'maximum entropy' distribution if I can't use the uniform distribution?

You basically answered your question, you can use the entropy of the distribution as a measure of randomness. For a Gaussian distribution, it can be computed directly from the standard standard deviation (and you could also directly use the standard deviation as a measure of randomness):

$$\operatorname{H}_{p}[X] = \operatorname{E}_{p}[-\ln p(x)] = \frac{1}{2}\ln\big(2\pi\sigma^2\big)+\frac{1}{2}$$

The KL from some distribution $q$ to a uniform distribution $p$ actually contains two terms, the negative entropy of the first distribution and the cross entropy between the two distributions. Because the log probability of an unbounded uniform distribution is constant, the cross entropy is a constant:

$$ \operatorname{KL}[q(x) \,\|\, p(x)] = \operatorname{E}_{q}[\ln q(x) - \ln p(x)] \\ = \operatorname{E}_{q}[\ln q(x)] - \operatorname{E}_{q}[\ln p(x)] \\ = -\operatorname{H}[q(x)] - \operatorname{E}_{q}[\mathrm{const}] \\ = -\operatorname{H}[q(x)] - \mathrm{const} $$

For categorical variables, the constant is the log probability of uniform categorical distribution, which is $\ln \frac{1}{N}=-\ln N$. For continuous variables, you can't have an unbounded uniform distribution because it would have an infinite normalizer. A trick is to instead consider an improper uniform distribution that is unnormalized, $p(x)\propto 1$.