13
$\begingroup$

Is there any use for the quantity $$ \int f(x)^2 dx $$ in statistics or information theory?

$\endgroup$
6
  • 5
    $\begingroup$ Renyi entropy $\endgroup$ – cardinal Apr 24 '11 at 15:46
  • $\begingroup$ $f$ is a pdf, right? $\endgroup$ – whuber Apr 24 '11 at 22:19
  • $\begingroup$ Yes, $f$ is a density. $\endgroup$ – charles.y.zheng Apr 25 '11 at 2:34
  • $\begingroup$ @cardinal Answer! $\endgroup$ – user88 Apr 25 '11 at 8:24
  • $\begingroup$ @mbq: Ok, I'll try to type something up a bit later that is worthy of an answer. :) $\endgroup$ – cardinal Apr 25 '11 at 12:05
24
$\begingroup$

Letting $f$ denote a probability density function (either with respect to Lebesgue or counting measure, respectively), the quantity $\newcommand{\rd}{\mathrm{d}}$ $$ H_\alpha(f) = -\frac{1}{\alpha-1} \log(\textstyle\int f^\alpha \rd \mu) $$ is known as the Renyi entropy of order $\alpha \geq 0$. It is a generalization of Shannon entropy that retains many of the same properties. For the case $\alpha = 1$, we interpret $H_1(f)$ as $\lim_{\alpha \to 1} H_{\alpha}(f)$, and this corresponds to the standard Shannon entropy $H(f)$.

Renyi introduced this in his paper

A. Renyi, On measures of information and entropy, Proc. 4th Berkeley Symp. on Math., Stat. and Prob. (1960), pp. 547–561.

which is well worth reading, not only for the ideas but for the exemplary exposition style.

The case $\alpha = 2$ is one of the more common choices for $\alpha$ and this special case is (also) often referred to as the Renyi entropy. Here we see that $$\newcommand{\e}{\mathbb{E}} H_2(f) = - \log( \textstyle\int f^2 \rd \mu ) = -\log( \e f(X) ) $$ for a random variable distributed with density $f$.

Note that $- \log(x)$ is a convex function and so, by Jensen's inequality we have $$ H_2(f) = -\log( \e f(X) ) \leq \e( -\log f(X) ) = - \e \log f(X) = H(f) $$ where the right-hand side denotes the Shannon entropy. Hence the Renyi entropy provides a lower bound for the Shannon entropy and, in many cases, is easier to calculate.

Another natural instance in which the Renyi entropy arises is when considering a discrete random variable $X$ and an independent copy $X^\star$. In some scenarios we want to know the probability that $X = X^\star$, which by an elementary calculation is $$\renewcommand{\Pr}{\mathbb{P}} \Pr(X = X^\star) = \sum_{i=1}^\infty \Pr(X = x_i, X^\star = x_i) = \sum_{i=1}^\infty \Pr(X = x_i) \Pr(X^\star = x_i) = e^{-H_2(f)} . $$

Here $f$ denotes the density with respect to counting measure on the set of values $\Omega = \{x_i: i \in \mathbb{N}\}$.

The (general) Renyi entropy is also apparently related to free energy of a system in thermal equilibrium, though I'm not personally up on that. A (very) recent paper on the subject is

J. C. Baez, Renyi entropy and free energy, arXiv [quant-ph] 1101.2098 (Feb. 2011).

$\endgroup$
3
  • $\begingroup$ I was indeed using the Renyi entropy as a substitute for the Shannon entropy; it is nice to see confirmation of my intuition. Thank you for the enlightening response. $\endgroup$ – charles.y.zheng Apr 26 '11 at 10:11
  • 1
    $\begingroup$ Many (but not all!) of the properties and usefulness of Shannon entropy arise from its convexity. If you look at the build up of the basics results in information theory, they more or less hinge on Jensen's inequality. So, in a certain (vague) sense, there is not too much that is (terribly) special about $-\log x$ as the particular nonlinearity that leads to a notion of "information". $\endgroup$ – cardinal Apr 26 '11 at 10:17
  • 1
    $\begingroup$ I see. Specifically, I need the property that the maximum entropy joint distribution which satisfies given marginals is the product of the marginals (what you would get from independence.) $\endgroup$ – charles.y.zheng Apr 26 '11 at 10:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.