# Negative Values in G test

I have a question, that regards the G-Test. This, as defined by Wikipedia to use $$G=2 \sum \limits_{i=0}^n O_i\ln(\frac{O_i}{E_i})$$ as a value to measure the observed samples $$O_i$$ of category $$i$$ with the expected oberservations $$E_i$$. Basically: The smaller this value, the better the model, that produced the $$E_i$$.

However, I have a question regarding this summation. As long, as we are over-observing ($$O_i>E_i$$) this will add up to a positive value. But there must be some categories with $$E_i>O_i$$, as $$\sum O_i = \sum E_i$$. These categories will have a negative effect on the total sum.

In my gullible mind, that has little to no experience with statistics, there can be a model, that totally mispredicts every observation, but still has $$G=0$$.

This problem is not to be found in the case of the Chi-Squared-Test, that will only add positive (squared) numbers.

I want to use the G-Test, as it is more suitable for categories with low occurrences. But these are more prone to under-observation, that leads back to the original problem.

Is there a clue, that I am missing? Has it to do with the equality of observed and expected outcomes, combined with the shape of the natural logarithm?

• Could you give an example that G is negative when you perform G test? Oct 15, 2018 at 18:07
• I don't know if $G$ can be negative as a whole. Can't make an example up on the spot. But there must be categories, that contribute to the total summation in a negative way. Oct 15, 2018 at 18:52
• I see. Maybe title should be "Negative components in G statistics for G test". Oct 15, 2018 at 20:42

Let's consider it informally. Conditioning on the total, $$E_i$$'s are fixed.

Where you're going wrong is missing that the downward effect from the negative terms is more than compensated for by the effect on the remaining terms.

Note that $$\ln(\frac{O_i}{E_i})$$ gets multiplied by $$O_i$$ so when you make the log term smaller (more negative) by dropping $$O_i$$ you also reduce what it gets multiplied by.

At the same time, when $$O_i$$ drops, other $$O$$'s will have to increase to compensate. So while this $$i$$-th term is multiplied now by a smaller number ($$O_i$$ dropped, after all) the other term is multiplied by a larger $$O$$ than before; that compensates for the drop in the other term - and then some.

The actual effect of increasing the overall discrepancy between $$O$$'s and $$E$$'s is to make the test statistic larger when considered across all the terms.

Let's take a very simple example - two categories with expected proportions of 60% and 40% in the two categories. Let's say we start with 6 in category 1 and 4 in category 2 (observed equals expected in both). The G statistic is 0.

So now holding n at 10, drop the 6 to a 5. The other observed count thereby increases to a 5 as well.

Now the G statistic is $$5\ln(5/6) + 5\ln(5/4) \approx 0.20$$

What if the other category had dropped instead? The 4 drops to a 3 and the 6 goes to a 7.

Now the G statistic is $$7\ln(7/6) + 3\ln(3/4) \approx 0.22$$

Either way, the negative contribution was more than compensated by the corresponding movement in another cell.

If you add more cells, it won't alter this outcome -- unambiguously increasing the overall discrepancy between $$O$$'s and $$E$$'s makes the statistic larger.

I understand your skepticism, but $$G$$ is always positive. This is a consequence of Jensen's inequality. Let $$N = \sum_i O_i = \sum_i E_i$$. $$G = 2N\sum_i \frac{O_i}{N} \log \frac{O_i/N}{E_i/N} = -2N \sum_i \pi(i) \log \frac{\gamma(i)}{\pi(i)},$$ where $$\pi(i)$$ and $$\gamma(i)$$ correspond to the estimated null and alternative distributions respectively. Now, let $$X$$ be a discrete random variable on $$\{1,\ldots,n\}$$ having probability $$\pi(i)$$ probability of $$X = i$$. Then by Jensen's inequality and the convexity of $$f(x) = -\log(x)$$, $$G = 2N \mathbb E\left[-\log\frac{\gamma(X)}{\pi(X)}\right] \ge -2N \log \mathbb E\frac{\gamma(X)}{\pi(X)}.$$ But $$\mathbb E \frac{\gamma(X)}{\pi(X)} = \sum_i \pi(i) \frac{\gamma(i)}{\pi(i)} = \sum_i \gamma(i) = 1,$$ so $$G \ge -2N \log(1) = 0$$.

• +1. However, in your first sentence you mean non-negative rather than positive -- your demonstration gives that it is $\geq 0$ and the bound is attainable; $G$ can actually be zero. For an example, see the initial case in the example in my answer which had $\pi = (0.6,0.4)^\prime$ and $O=(6,4)^\prime$), so $G=0$. Oct 16, 2018 at 22:56