Why does discrete data distribution has differential entropy of negative infinity? Recently I've been reading a paper.
In section 3.1, it says "Since the discrete data distribution has differential entropy of negative infinity, this can lead to arbitrary high likelihoods even on test data."
I don't understand why could differential entropy be 'negative infinity', and why can it cause 'high likelihood'.
Can someone help me?
 A: The differential entropy is defined as:
$$h=-\int_{-\infty}^\infty f(x)\ln(f(x))dx$$
vwhere $f(x)$ is the PDf of the function. For the discrete distribution, there is not PDF, but there is a probability mass function. However, we could represent the PDF with Dirac delta function $\delta(x)$:
$$f(x)=\sum_ip_i\delta(x-x_i)$$
where $p_i$ is the probability of outcome $x_i$.
For instance, Benroulli distribution psudo PDF would be: $$f(x)=\frac{\delta(x)+\delta(x-1)}{2}$$
You can see that this sort of behaves like PDF:
$$\int_{-\infty}^\infty f(x)dx=\sum_ip_i\equiv 1$$ etc.
Now, if you plug this PDF like creature into the differential entropy you get the following:
$$h=-\int_{-\infty}^\infty f(x)\ln(f(x))dx=
-\int_{-\infty}^\infty \left(\sum_ip_i\delta(x-x_i)\right)\ln\left(\sum_kp_k\delta(x-x_k)\right)dx=\\
-\sum_ip_i\ln\left(\sum_kp_k\delta(x_i-x_k)\right)=-\infty$$
You can see why this doesn't converge: for any $k= i$ you get $\delta(0)$, which can be interpreted as infinity.
Hence, the conclusion that differential entropy for discrete distributions is negative infinity. However, you should not be normally dealing with this, because for discrete distribution you would usually apply a conventional entropy formula : $h=-\sum_i p_i\ln p_i$. 
A: The next sentence in that paper has a citation that clears it up:

To avoid this case, it is becoming best practice to add real-valued noise
to the integer pixel values to dequantize the data (e.g., Uria et al., 2013; van den Oord & Schrauwen, 2014; Theis & Bethge, 2015))

The Uria et al. 2013 paper on RNADE explains it well:

Pixels in this dataset can take a finite number of brightness values ranging from 0 to 255. Modeling discretized data using a real-valued distribution can lead to arbitrarily high density values, by locating
narrow high density spike on each of the possible discrete values. In order to avoid this "cheating" solution, we added noise uniformly distributed between 0 and 1 to the value of each pixel.

