According to Wikipedia, the G-test statistic is "proportional to the Kullback–Leibler divergence of the empirical distribution from the theoretical distribution." To get the relationship between $ G $ and the KL divergence: $$ \begin{align*} G &= 2 \sum_i O_i \ln\left(\frac{O_i}{E_i}\right) \\ &= 2 \sum_i n P(i) \ln\left(\frac{n P(i)}{n Q(i)}\right) \\ &= 2n \sum_i P(i) \ln\left(\frac{P(i)}{Q(i)}\right) \\ &= 2n \times D_{KL}(P \| Q) \end{align*} $$

This is where I'm a little confused. $ D_{KL}(P \| Q) $ represents the divergence of $ Q $ (the expected distribution) from $ P $ (the observed distribution). This is the opposite of the statement in the Wikipedia page (assuming that "theoretical" refers to expected and "empirical" refers to observed).

Now from an information theoretic interpretation of KL divergence, $ P $ is represents the "true" distribution and $ Q $ is the "approximation". But from this derivation, the result seems counter-intuitive. Shouldn't the expected distribution correspond to the true distribution and the observed distribution correspond to the approximation?


People use inconsistent language with the KL divergence; sometimes "the divergence of $Q$ from $P$" means $KL(P \| Q)$; sometimes it means $KL(Q \| P)$.

$KL(\text{true} \| \text{approximation})$ has a nice information-theoretic interpretation as the inefficiency due to encoding the true distribution with a code built based on the approximation. But that doesn't mean that $KL(\text{approximation} \| \text{true})$ doesn't ever come up; it does all the time. In this case, you could think of it as the loss of encoding your data with a code based on the expected distribution if you were looking for an info-theory interpretation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.