Recently I've been reading a paper. In section 3.1, it says "Since the discrete data distribution has differential entropy of negative infinity, this can lead to arbitrary high likelihoods even on test data."

I don't understand why does differential entropy could be 'negative infinity', and why does it can cause 'high likelihood'.

Can someone help me?

  • $\begingroup$ it's probably an unfortunate language: it can be negative for discrete distributions but doesn't have to be negative infinity $\endgroup$ – Aksakal May 20 '19 at 19:35
  • $\begingroup$ @Aksakal The paper refers explicitly to the differential entropy between some underlying continuous distribution and a discrete distribution: "an image is treated as an instance of a continuous random variable." $\endgroup$ – whuber May 20 '19 at 19:43
  • $\begingroup$ @whuber maybe "the" refers to a particular distribution, but the language appears to suggest a general statement about the discrete distributions. also, i think the paper refers to a differential entropy extension of shannon's discrete entropy, not the difference between distributions as it's evident from Eq(3) $\endgroup$ – Aksakal May 20 '19 at 19:47
  • $\begingroup$ @Aksakal. Right: and isn't it the case that in that sense all continuous distributions are infinitely far from all non-continuous distributions? $\endgroup$ – whuber May 20 '19 at 20:02

The differential entropy is defined as: $$h=-\int_{-\infty}^\infty f(x)\ln(f(x))dx$$ vwhere $f(x)$ is the PDf of the function. For the discrete distribution, there is not PDF, but there is a probability mass function. However, we could represent the PDF with Dirac delta function $\delta(x)$: $$f(x)=\sum_ip_i\delta(x-x_i)$$ where $p_i$ is the probability of outcome $x_i$. For instance, Benroulli distribution psudo PDF would be: $$f(x)=\frac{\delta(x)+\delta(x-1)}{2}$$

You can see that this sort of behaves like PDF: $$\int_{-\infty}^\infty f(x)dx=\sum_ip_i\equiv 1$$ etc.

Now, if you plug this PDF like creature into the differential entropy you get the following: $$h=-\int_{-\infty}^\infty f(x)\ln(f(x))dx= -\int_{-\infty}^\infty \left(\sum_ip_i\delta(x-x_i)\right)\ln\left(\sum_kp_k\delta(x-x_k)\right)dx=\\ -\sum_ip_i\ln\left(\sum_kp_k\delta(x_i-x_k)\right)=-\infty$$ You can see why this doesn't converge: for any $k= i$ you get $\delta(0)$, which can be interpreted as infinity.

Hence, the conclusion that differential entropy for discrete distributions is negative infinity. However, you should not be normally dealing with this, because for discrete distribution you would usually apply a conventional entropy formula : $h=-\sum_i p_i\ln p_i$.

  • $\begingroup$ Thanks for the answer! I'm curious about how we can represent the PDF as a sum of Dirac delta's? Do you perhaps have a source or further explanation? Thank you! $\endgroup$ – Christabella Irwanto Jan 12 at 17:56
  • 1
    $\begingroup$ @ChristabellaIrwanto, Bernoulli distribution PDF is $f(x)=\frac 1 2\delta(0)+\frac 1 2\delta(1)$. CDF is $F(x)=\int_{-\infty}^xf(s)ds$ so $F(0)=0,F(0+)=1/2,F(1)=1/2,F(1+)=1$ $\endgroup$ – Aksakal Jan 14 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.