Is there any use for the quantity $\int f(x)^2 dx$ in statistics or information theory?

Question

Is there any use for the quantity $$ \int f(x)^2 dx $$ in statistics or information theory?

Answer 1

Letting $f$ denote a probability density function (either with respect to Lebesgue or counting measure, respectively), the quantity $\newcommand{\rd}{\mathrm{d}}$ $$ H_\alpha(f) = -\frac{1}{\alpha-1} \log(\textstyle\int f^\alpha \rd \mu) $$ is known as the Renyi entropy of order $\alpha \geq 0$. It is a generalization of Shannon entropy that retains many of the same properties. For the case $\alpha = 1$, we interpret $H_1(f)$ as $\lim_{\alpha \to 1} H_{\alpha}(f)$, and this corresponds to the standard Shannon entropy $H(f)$.

Renyi introduced this in his paper

A. Renyi, On measures of information and entropy, Proc. 4th Berkeley Symp. on Math., Stat. and Prob. (1960), pp. 547–561.

which is well worth reading, not only for the ideas but for the exemplary exposition style.

The case $\alpha = 2$ is one of the more common choices for $\alpha$ and this special case is (also) often referred to as the Renyi entropy. Here we see that $$\newcommand{\e}{\mathbb{E}} H_2(f) = - \log( \textstyle\int f^2 \rd \mu ) = -\log( \e f(X) ) $$ for a random variable distributed with density $f$.

Note that $- \log(x)$ is a convex function and so, by Jensen's inequality we have $$ H_2(f) = -\log( \e f(X) ) \leq \e( -\log f(X) ) = - \e \log f(X) = H(f) $$ where the right-hand side denotes the Shannon entropy. Hence the Renyi entropy provides a lower bound for the Shannon entropy and, in many cases, is easier to calculate.

Another natural instance in which the Renyi entropy arises is when considering a discrete random variable $X$ and an independent copy $X^\star$. In some scenarios we want to know the probability that $X = X^\star$, which by an elementary calculation is $$\renewcommand{\Pr}{\mathbb{P}} \Pr(X = X^\star) = \sum_{i=1}^\infty \Pr(X = x_i, X^\star = x_i) = \sum_{i=1}^\infty \Pr(X = x_i) \Pr(X^\star = x_i) = e^{-H_2(f)} . $$

Here $f$ denotes the density with respect to counting measure on the set of values $\Omega = \{x_i: i \in \mathbb{N}\}$.

The (general) Renyi entropy is also apparently related to free energy of a system in thermal equilibrium, though I'm not personally up on that. A (very) recent paper on the subject is

J. C. Baez, Renyi entropy and free energy, arXiv [quant-ph] 1101.2098 (Feb. 2011).