14
$\begingroup$

Is there any use for the quantity $$ \int f(x)^2 dx $$ in statistics or information theory? $f$ is a probability density.

$\endgroup$
6
  • 6
    $\begingroup$ Renyi entropy $\endgroup$
    – cardinal
    Apr 24, 2011 at 15:46
  • 1
    $\begingroup$ $f$ is a pdf, right? $\endgroup$
    – whuber
    Apr 24, 2011 at 22:19
  • $\begingroup$ Yes, $f$ is a density. $\endgroup$ Apr 25, 2011 at 2:34
  • $\begingroup$ @cardinal Answer! $\endgroup$
    – user88
    Apr 25, 2011 at 8:24
  • $\begingroup$ @mbq: Ok, I'll try to type something up a bit later that is worthy of an answer. :) $\endgroup$
    – cardinal
    Apr 25, 2011 at 12:05

1 Answer 1

26
$\begingroup$

Letting $f$ denote a probability density function (either with respect to Lebesgue or counting measure, respectively), the quantity $\newcommand{\rd}{\mathrm{d}}$ $$ H_\alpha(f) = -\frac{1}{\alpha-1} \log(\textstyle\int f^\alpha \rd \mu) $$ is known as the Renyi entropy of order $\alpha \geq 0$. It is a generalization of Shannon entropy that retains many of the same properties. For the case $\alpha = 1$, we interpret $H_1(f)$ as $\lim_{\alpha \to 1} H_{\alpha}(f)$, and this corresponds to the standard Shannon entropy $H(f)$.

Renyi introduced this in his paper

A. Renyi, On measures of information and entropy, Proc. 4th Berkeley Symp. on Math., Stat. and Prob. (1960), pp. 547–561.

which is well worth reading, not only for the ideas but for the exemplary exposition style.

The case $\alpha = 2$ is one of the more common choices for $\alpha$ and this special case is (also) often referred to as the Renyi entropy. Here we see that $$\newcommand{\e}{\mathbb{E}} H_2(f) = - \log( \textstyle\int f^2 \rd \mu ) = -\log( \e f(X) ) $$ for a random variable distributed with density $f$.

Note that $- \log(x)$ is a convex function and so, by Jensen's inequality we have $$ H_2(f) = -\log( \e f(X) ) \leq \e( -\log f(X) ) = - \e \log f(X) = H(f) $$ where the right-hand side denotes the Shannon entropy. Hence the Renyi entropy provides a lower bound for the Shannon entropy and, in many cases, is easier to calculate.

Another natural instance in which the Renyi entropy arises is when considering a discrete random variable $X$ and an independent copy $X^\star$. In some scenarios we want to know the probability that $X = X^\star$, which by an elementary calculation is $$\renewcommand{\Pr}{\mathbb{P}} \Pr(X = X^\star) = \sum_{i=1}^\infty \Pr(X = x_i, X^\star = x_i) = \sum_{i=1}^\infty \Pr(X = x_i) \Pr(X^\star = x_i) = e^{-H_2(f)} . $$

Here $f$ denotes the density with respect to counting measure on the set of values $\Omega = \{x_i: i \in \mathbb{N}\}$.

The (general) Renyi entropy is also apparently related to free energy of a system in thermal equilibrium, though I'm not personally up on that. A (very) recent paper on the subject is

J. C. Baez, Renyi entropy and free energy, arXiv [quant-ph] 1101.2098 (Feb. 2011).

$\endgroup$
3
  • 1
    $\begingroup$ I was indeed using the Renyi entropy as a substitute for the Shannon entropy; it is nice to see confirmation of my intuition. Thank you for the enlightening response. $\endgroup$ Apr 26, 2011 at 10:11
  • 1
    $\begingroup$ Many (but not all!) of the properties and usefulness of Shannon entropy arise from its convexity. If you look at the build up of the basics results in information theory, they more or less hinge on Jensen's inequality. So, in a certain (vague) sense, there is not too much that is (terribly) special about $-\log x$ as the particular nonlinearity that leads to a notion of "information". $\endgroup$
    – cardinal
    Apr 26, 2011 at 10:17
  • 1
    $\begingroup$ I see. Specifically, I need the property that the maximum entropy joint distribution which satisfies given marginals is the product of the marginals (what you would get from independence.) $\endgroup$ Apr 26, 2011 at 10:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.