Negative Values in G test I have a question, that regards the G-Test. 
This, as defined by Wikipedia to use 
$$
G=2 \sum \limits_{i=0}^n O_i\ln(\frac{O_i}{E_i})
$$
as a value to measure the observed samples $O_i$ of category $i$ with the expected oberservations $E_i$. 
Basically: The smaller this value, the better the model, that produced the $E_i$. 
However, I have a question regarding this summation. As long, as we are over-observing ($O_i>E_i$) this will add up to a positive value. But there must be some categories with $E_i>O_i$, as $\sum O_i = \sum E_i$. These categories will have a negative effect on the total sum. 
In my gullible mind, that has little to no experience with statistics, there can be a model, that totally mispredicts every observation, but still has $G=0$. 
This problem is not to be found in the case of the Chi-Squared-Test, that will only add positive (squared) numbers. 
I want to use the G-Test, as it is more suitable for categories with low occurrences. But these are more prone to under-observation, that leads back to the original problem. 
Is there a clue, that I am missing? Has it to do with the equality of observed and expected outcomes, combined with the shape of the natural logarithm?
 A: Let's consider it informally. Conditioning on the total, $E_i$'s are fixed. 
Where you're going wrong is missing that the downward effect from the negative terms is more than compensated for by the effect on the remaining terms. 
Note that $\ln(\frac{O_i}{E_i})$ gets multiplied by $O_i$ so when you make the log term smaller (more negative) by dropping $O_i$ you also reduce what it gets multiplied by.
At the same time, when $O_i$ drops, other $O$'s will have to increase to compensate. So while this $i$-th term is multiplied now by a smaller number ($O_i$ dropped, after all) the other term is multiplied by a larger $O$ than before; that compensates for the drop in the other term - and then some.
The actual effect of increasing the overall discrepancy between $O$'s and $E$'s is to make the test statistic larger when considered across all the terms.
Let's take a very simple example - two categories with expected proportions of 60% and 40% in the two categories. Let's say we start with 6 in category 1 and 4 in category 2 (observed equals expected in both). The G statistic is 0.
So now holding n at 10, drop the 6 to a 5. The other observed count thereby increases to a 5 as well.
Now the G statistic is $5\ln(5/6) + 5\ln(5/4) \approx 0.20$
What if the other category had dropped instead? The 4 drops to a 3 and the 6 goes to a 7.
Now the G statistic is $7\ln(7/6) + 3\ln(3/4) \approx 0.22$
Either way, the negative contribution was more than compensated by the corresponding movement in another cell.
If you add more cells, it won't alter this outcome -- unambiguously increasing the overall discrepancy between $O$'s and $E$'s makes the statistic larger.
A: I understand your skepticism, but $G$ is always positive. This is a consequence of Jensen's inequality.  Let $N = \sum_i O_i = \sum_i E_i$. 
$$
G
= 
2N\sum_i \frac{O_i}{N} \log \frac{O_i/N}{E_i/N}
= 
-2N \sum_i \pi(i) \log \frac{\gamma(i)}{\pi(i)},
$$
where $\pi(i)$ and $\gamma(i)$ correspond to the estimated null and alternative distributions respectively. Now, let $X$ be a discrete random variable on $\{1,\ldots,n\}$ having probability $\pi(i)$ probability of $X = i$. Then by Jensen's inequality and the convexity of $f(x) = -\log(x)$,
$$
G = 2N \mathbb E\left[-\log\frac{\gamma(X)}{\pi(X)}\right]
\ge
-2N \log \mathbb E\frac{\gamma(X)}{\pi(X)}. 
$$
But 
$$
\mathbb E \frac{\gamma(X)}{\pi(X)}
= 
\sum_i \pi(i) \frac{\gamma(i)}{\pi(i)} = \sum_i \gamma(i) = 1,
$$
so $G \ge -2N \log(1) = 0$. 
