Is the KL divergence not defined because uniform has bounded support and gaussian has unbounded support?

How else can I calculate the distance of my gaussian to a 'maximum entropy' distribution if I can't use the uniform distribution?

  • 1
    $\begingroup$ The distance is infinite. $\endgroup$ – Xi'an May 21 '19 at 6:28

The KL divergence $$ KL\left(P \middle\| Q\right) = \int \log \frac{d P}{d Q }dP $$ is only defined if the Radon-Nikodym derivative exists, which is when $P$ is absolutely continuous with respect to $Q$ (written $P \ll Q$). This means that there can't be any sets $A$ where $P(A) > 0$ and $Q(A) = 0$, otherwise we would be dividing by zero.

In your case, $p$ is the density of the uniform random variable, and $q$ is the density of the normal random variable (they are both dominated by the Lebesgue measure), so you could calculate $$ KL\left(P \middle\| Q\right) = \int \log \frac{p(x)}{q(x)}p(x)dx, $$ but you couldn't calculate $KL\left(Q \middle\| P\right)$. You can calculate $KL\left(P \middle\| Q\right)$ because there are no sets $A$ such that $\int_A p(x) dx > 0$ and $\int_A q(x) dx = 0$.

| cite | improve this answer | |
  • $\begingroup$ Yes, unfortunately I need KL(Q||P). j $\endgroup$ – user3180 May 21 '19 at 4:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.