Information theoretic central limit theorem

Question

The simplest form of the information theoretic CLT is the following:

Let $X_1, X_2,\dots$ be iid with mean $0$ and variance $1$. Let $f_n$ be the density of the normalized sum $\frac{\sum_{i=1}^n X_i}{\sqrt{n}}$ and $\phi$ be the standard Gaussian density. Then the information theoretic CLT states that, if $D(f_n\|\phi)=\int f_n \log(f_n/\phi) dx$ is finite for some $n$, then $D(f_n\|\phi)\to 0$ as $n\to \infty$.

Certainly this convergence, in a sense, is "stronger" than the well establised convergences in the literature, convergence in distribution and convergence in $L_1$-metric, thanks to Pinsker's inequality $\left(\int |f_n-\phi|\right)^2\le 2\cdot \int f_n \log(f_n/\phi)$. That is, convergence in KL-divergence implies convergence in distribution and convergence in $L_1$ distance.

I would like to know two things.

1. What is so great about the result $D(f_n\|\phi)\to 0$?

2. Is it just because of the reason stated in the third paragraph we say convergence in KL-divergence (i.e., $D(f_n\|\phi)\to 0$) is stronger?

NB: I asked this question sometime ago in math.stackexchange where I didn't get any answer.

Answer 1

One thing which is great with this theorem is that it suggests limit theorems in some settings where the usual central limit theorem do not apply. For instance, in situations where the maximum entropy-distribution is some nonnormal distribution, such as for distributions on the circle, it suggests convergence to a uniform distribution.

Answer 2

After looking around, I could not find any example of convergence in distribution without convergence in relative entropy, so this is hard to measure the "greatness" of that result.

To me, it looks like this result simply describes the relative entropy of convolution products. It is often viewed as an alternative interpretation and proof framework of the Central Limit Theorem, and I am not sure it has a direct implication in probability theory (even though it does in information theory).

From Information Theory and the Central Limit Theorem (page 19).

The Second Law of Thermodynamics states that the thermodynamic entropy always increases with time, implying some kind of convergence to the Gibbs state. Conservation of energy means that $E$ remains constant during this time evolution, so we can tell from the start which Gibbs state will be the limit.
We will regard the Central Limit Theorem in the same way, by showing that the information-theoretic entropy increases to its maximum as we take convolutions, implying convergence to the Gaussian. Normalizing appropriately means that the variance remains constant during convolutions so we can tell from the start which Gaussian will be the limit.

Answer 3

$D(f_n\|\phi)\rightarrow 0$ assures that there's no "distance" between the distribution of the sum of random variables and the gaussian density as $n\rightarrow\infty$ just because of the definition of K-L divergence, so it's the proof itself. Perhaps I misunderstood your question.

About the second point as you appointed, it's responded in your paragraph.