Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

The simplest form of the information theoretic CLT is the following:

Let $X_1, X_2,\dots$ be iid with mean $0$ and variance $1$. Let $f_n$ be the density of the normalized sum $\frac{\sum_{i=1}^n X_i}{\sqrt{n}}$ and $\phi$ be the standard Gaussian density. Then the information theoretic CLT states that, if $D(f_n\|\phi)=\int f_n \log(f_n/\phi) dx$ is finite for some $n$, then $D(f_n\|\phi)\to 0$ as $n\to \infty$.

Certainly this convergence, in a sense, is "stronger" than the well establised convergences in the literature, convergence in distribution and convergence in $L_1$-metric, thanks to Pinsker's inequality $\left(\int |f_n-\phi|\right)^2\le 2\cdot \int f_n \log(f_n/\phi)$. That is, convergence in KL-divergence implies convergence in distribution and convergence in $L_1$ distance.

I would like to know two things.

  1. What is so great about the result $D(f_n\|\phi)\to 0$?

  2. Is it just because of the reason stated in the third paragraph we say convergence in KL-divergence (i.e., $D(f_n\|\phi)\to 0$) is stronger?

Thanks.

NB: I asked this question sometime ago in math.stackexchange where I didn't get any answer.

share|improve this question
    
Please provide a link to the duplicate math.SE question. –  cardinal Jun 28 '12 at 13:32
3  
Your statement seems to implicitly assume the existence of a density (with respect to Lebesgue measure). You may be interested in this short and delightful paper: A. R. Barron (1986), Entropy and the Central Limit Theorem Ann. Probab., vol 14, no. 1, 336-342. (open access). –  cardinal Jun 28 '12 at 13:36
1  
I had looked at that paper already. He has given a motivation in information theoretic perspective in the second paragraph of page 1. It was not all that clear to me at that time. Now it looks ok. Still, if one can explain the following clearly and post as an answer, it would be great. "From information theory, the relative entropy $D_n$ is the least upper bound to the redundancy (excess average description length) of the Shannon code based on the normal distribution when describing quantizations of samples from $f_n$." I have deleted that question in math.SE as it didn't attract anybody there –  Ashok Jun 28 '12 at 14:02
    
@cardinal: tks for the nice paper. –  Zen Jun 28 '12 at 18:05
add comment

3 Answers

After looking around, I could not find any example of convergence in distribution without convergence in relative entropy, so this is hard to measure the "greatness" of that result.

To me, it looks like this result simply describes the relative entropy of convolution products. It is often viewed as an alternative interpretation and proof framework of the Central Limit Theorem, and I am not sure it has a direct implication in probability theory (even though it does in information theory).

From Information Theory and the Central Limit Theorem (page 19).

The Second Law of Thermodynamics states that the thermodynamic entropy always increases with time, implying some kind of convergence to the Gibbs state. Conservation of energy means that $E$ remains constant during this time evolution, so we can tell from the start which Gibbs state will be the limit.
We will regard the Central Limit Theorem in the same way, by showing that the information-theoretic entropy increases to its maximum as we take convolutions, implying convergence to the Gaussian. Normalizing appropriately means that the variance remains constant during convolutions so we can tell from the start which Gaussian will be the limit.

share|improve this answer
    
There are lots of examples of convergence in distribution without convergence in relative entropy -- any time the $X_i$ have a discrete distribution and the CLT applies. –  Mark Meckes Aug 27 '12 at 19:36
add comment

$D(f_n\|\phi)\rightarrow 0$ assures that there's no "distance" between the distribution of the sum of random variables and the gaussian density as $n\rightarrow\infty$ just because of the definition of K-L divergence, so it's the proof itself. Perhaps I misunderstood your question.

About the second point as you appointed, it's responded in your paragraph.

share|improve this answer
add comment

One thing which is great with this theorem is that it suggests limit theorems in some settings where the usual central limit theorem do not apply. For instance, in situations where the maximum entropy-distribution is , such as for distributions on the circle, it suggests convergence to a uniform distribution.

share|improve this answer
    
I don't understand. As I already mentioned, convergence in KL divergence implies convergence in distribution, know? So whereever information theoretic CLT applies, usual CLT also applies. Moreover, information theoretic CLT also assumes finite variance. Or am I missing something? –  Ashok Oct 1 '12 at 8:02
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.