If $f(x)=\frac{1}{x\ln{x}^2}, x\ge e$, $h(X)=+\infty$. But if I hope to let $h(X)=-\infty$, can I find such a function $f(x)$ without using limit and delta function?

  • 1
    $\begingroup$ Did you try constructing a series of uniform distributions with changing heights and widths? $\endgroup$ Feb 2 '16 at 13:52

There's no avoiding some kind of limit, because the differentiable entropy itself is defined in terms of an integral (a limit) and the very meaning of "$=\infty$" refers to a diverging sequence. But let's see what we can do with a minimum of fuss.

For $n=0,1,2,3,\ldots,$ let

$$x(n) = \sum_{i=1}^n \frac{e^{-i}}{i^{3/2}}.$$

For each $n\ge 1$ and $x(n-1) \lt t \le x(n)$ define

$$f_0(t) = e^n$$

and otherwise define $f_0(t) = 0.$

Compute that

$$C = \int_\mathbb{R} f_0(t)\,\mathrm{d}t =\sum_{n=1}^\infty e^{n}\frac{e^{-n}}{n^{3/2}}\lt \infty.$$

(In fact, $C=\zeta(3/2)\approx 2.6124.$)

Since $f_0 \ge 0$ everywhere and is obviously measurable, the function

$$f(t) = \frac{1}{C} f_0(t)$$

is a probability density function. (Alas, $f$ is no fun to plot: it rises incredibly rapidly as its argument approaches $0.4284407\ldots$ and so looks just like a spike there. But the spike gets so skinny so quickly that it still has finite area. It's not a delta function, though: $f$ is a genuine function, as you can see from its explicit construction.)

The entropy is

$$-H(f) = \int_{\mathbb{R}}\log(f(t))f(t)\,\mathrm{d}t = \log(C)+\frac{1}{C}\sum_{n=1}^\infty n e^n\frac{e^{-n}}{n^{3/2}} = \log(C)+\frac{1}{C}\sum_{n=1}^\infty\frac{1}{n^{1/2}},$$

which diverges (compare it to the integral of $x^{-1/2}\mathrm{d}x$): that's what it means for the (negative of the) entropy to be infinite.


This $f$ is a step function, which makes it discontinuous. The construction can be modified to make $f$ continuous--or even continuously differentiable of any order--by using a probability density function $g$ defined on $[0,1]$ which is sufficiently differentiable at the endpoints $0$ and $1.$ (A Beta$(a,a)$ distribution for large $a$ will work.) In the construction multiply $f_0$ by a scale, shifted version of $g.$ That is, replace $f_0$ by the function

$$f_0(t) = e^n\, \frac{1}{x(n) - x(n-1)} g\left(\frac{t - x(n)}{x(n)-x(n-1)}\right).$$

Exploit the inequality $x(n) - x(n-1)\lt 1$ (for $n\gt 1$) to establish that the measure of the set $f_0^{-1}[e^n,\infty)$ relative to the interval $x(n)-x(n-1)$ exceeds the measure $\pi$ of the set $g^{-1}[1,\infty),$ which must be positive. This easily shows the summation in the entropy calculation (for the modified construction) exceeds $\pi$ times the summation in the original calculation, whence the modified entropy diverges too.

One problem remains: At the point $\lim_{n\to\infty} x(n),$ where the spike soars to infinity, $f$ cannot be continuous. Address this with an additional modification: instead of defining $f_0$ on the intervals $[x(n-1),x(n)],$ define it on the intervals $[n+x(n-1), n+x(n)].$ This just pushes the probability around and thereby doesn't change the entropy at all. However, it also pushes the problem at the spike off to $+\infty.$ The graph of $f$ looks like an infinite comb with teeth of exponentially increasing heights (and more than exponentially decreasing widths): see the figure. With the preceding modification, each tooth is continuous and merges continuously with the $0$ values between the teeth.

The figure shows the effects of this second modification on the original step function and a smoothed version. Note the semilogarithmic scale.



To augment the wonderful answer given by whuber, it is worth noting an additional aspect of dealing with continuous functions defined on the real numbers or subsets of the real numbers. Given a function of this kind, one can define an "extension" function on the closure of the domain (which is a subset of the extended real numbers) which explicitly gives each boundary value a function output equivalent to its limit value. This allows you to work with a function where the limit values are defined explicitly. It also gives a representation of the function that is closer to how these things look in mathematical/statistical computing platforms, where there are explicit values representing positive and negative infinity.

How does this work: Suppose you have a continuous function $H: \mathcal{D} \rightarrow \mathbb{R}$ that is defined on some subset of the real numbers. Continuity of the function means that limits of the function are defined at the boundary points of the domain $\mathcal{D}$ (with some limits possibly being infinite). It is natural to define an extension function on the set of extended real numbers as follows. Define the extension function $H_*: \text{closure}(\mathcal{D}) \rightarrow \bar{\mathbb{R}}$ by:

$$H_*(x) = \lim_{a \rightarrow x} H(a).$$

For all $x \in \mathcal{D}$ we have $H_*(x) = \lim_{a \rightarrow x} H(a) = H(x)$ (which follows from continuity of the function), but the function is also defined at values on the boundary $\text{closure}(\mathcal{D}) - \mathcal{D}$. In this way, one can now write limits of the function $H$ explicitly with the function $H_*$.

An example: Consider the real version of the logarithm function $\log: (0, \infty) \rightarrow \mathbb{R}$. This is a continuous function with well-known limits at the boundaries, so we can extend it to $\log_* : [0, \infty] \rightarrow \bar{\mathbb{R}}$ as:

$$\log_*(x) = \begin{cases} -\infty & & & \text{for } x = 0, \\[6pt] \log(x) & & & \text{for } 0 < x < \infty, \\[6pt] +\infty & & & \text{for } x = \infty. \\[6pt] \end{cases}$$

Most mathematical/statistical software has explicit elements for positive and negative infinity, so it gives you the extended version of the logarithm function --- i.e., you can put in the inputs zero or positive infinity and you will get the outputs negative or positive infinity. Indeed, the extended logarithm function used here is the one used in statistical software. For example, using R you get:

[1] -Inf

[1] Inf

Note: For explanatory convenience, I have denoted the extended functions using different notation than the initial functions, but in practice they are usually denoted with the same notation. For example, the extended logarithm function would usually just be denoted as $\log$ irrespective of its domain.

  • $\begingroup$ This is interesting, but for computing the entropy don't we really need an extension of the function $x\to x\log(x)$? In that case, its value ought to be zero at $x=0.$ I am concerned that this approach loses the subtleties of analyzing the entropy. Please note, too, that the density defined in my answer is everywhere finite: no infinities are involved. (Although the density function itself is not continuous, it is possible to modify it into one that is continuous--and still finite!--everywhere. I have edited my answer to sketch such a modification.) $\endgroup$
    – whuber
    Feb 3 at 14:45
  • $\begingroup$ Indeed --- the $\log$ example I used was just for illustration. Honestly, I'm not really sure if this material is applicable here, but I gave it because the title of the question suggests that the questioner is interested in representing functions that take infinite values. $\endgroup$
    – Ben
    Feb 3 at 21:13
  • $\begingroup$ My best guess--at which I arrived after posting my analysis--is that one can represent a delta distribution as a weak limit of densities converging to a point and then try to extend the definition of $H$ via continuity to generalized functions, and in this sense can construct examples with $H=-\infty;$ but the OP wishes to avoid such constructions in favor of an actual density (that is, a distribution that is absolutely continuous w.r.t. Lebesgue measure) if possible. Such distribution functions can equal $+\infty$ only on a set of measure zero. $\endgroup$
    – whuber
    Feb 3 at 21:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.