1
$\begingroup$

I have a question, that regards the G-Test. This, as defined by Wikipedia to use $$ G=2 \sum \limits_{i=0}^n O_i\ln(\frac{O_i}{E_i}) $$ as a value to measure the observed samples $O_i$ of category $i$ with the expected oberservations $E_i$. Basically: The smaller this value, the better the model, that produced the $E_i$.

However, I have a question regarding this summation. As long, as we are over-observing ($O_i>E_i$) this will add up to a positive value. But there must be some categories with $E_i>O_i$, as $\sum O_i = \sum E_i$. These categories will have a negative effect on the total sum.

In my gullible mind, that has little to no experience with statistics, there can be a model, that totally mispredicts every observation, but still has $G=0$.

This problem is not to be found in the case of the Chi-Squared-Test, that will only add positive (squared) numbers.

I want to use the G-Test, as it is more suitable for categories with low occurrences. But these are more prone to under-observation, that leads back to the original problem.

Is there a clue, that I am missing? Has it to do with the equality of observed and expected outcomes, combined with the shape of the natural logarithm?

$\endgroup$
3
  • $\begingroup$ Could you give an example that G is negative when you perform G test? $\endgroup$
    – user158565
    Oct 15, 2018 at 18:07
  • $\begingroup$ I don't know if $G$ can be negative as a whole. Can't make an example up on the spot. But there must be categories, that contribute to the total summation in a negative way. $\endgroup$
    – Laray
    Oct 15, 2018 at 18:52
  • $\begingroup$ I see. Maybe title should be "Negative components in G statistics for G test". $\endgroup$
    – user158565
    Oct 15, 2018 at 20:42

2 Answers 2

1
$\begingroup$

Let's consider it informally. Conditioning on the total, $E_i$'s are fixed.

Where you're going wrong is missing that the downward effect from the negative terms is more than compensated for by the effect on the remaining terms.

Note that $\ln(\frac{O_i}{E_i})$ gets multiplied by $O_i$ so when you make the log term smaller (more negative) by dropping $O_i$ you also reduce what it gets multiplied by.

At the same time, when $O_i$ drops, other $O$'s will have to increase to compensate. So while this $i$-th term is multiplied now by a smaller number ($O_i$ dropped, after all) the other term is multiplied by a larger $O$ than before; that compensates for the drop in the other term - and then some.

The actual effect of increasing the overall discrepancy between $O$'s and $E$'s is to make the test statistic larger when considered across all the terms.

Let's take a very simple example - two categories with expected proportions of 60% and 40% in the two categories. Let's say we start with 6 in category 1 and 4 in category 2 (observed equals expected in both). The G statistic is 0.

So now holding n at 10, drop the 6 to a 5. The other observed count thereby increases to a 5 as well.

Now the G statistic is $5\ln(5/6) + 5\ln(5/4) \approx 0.20$

What if the other category had dropped instead? The 4 drops to a 3 and the 6 goes to a 7.

Now the G statistic is $7\ln(7/6) + 3\ln(3/4) \approx 0.22$

Either way, the negative contribution was more than compensated by the corresponding movement in another cell.

If you add more cells, it won't alter this outcome -- unambiguously increasing the overall discrepancy between $O$'s and $E$'s makes the statistic larger.

$\endgroup$
1
$\begingroup$

I understand your skepticism, but $G$ is always positive. This is a consequence of Jensen's inequality. Let $N = \sum_i O_i = \sum_i E_i$. $$ G = 2N\sum_i \frac{O_i}{N} \log \frac{O_i/N}{E_i/N} = -2N \sum_i \pi(i) \log \frac{\gamma(i)}{\pi(i)}, $$ where $\pi(i)$ and $\gamma(i)$ correspond to the estimated null and alternative distributions respectively. Now, let $X$ be a discrete random variable on $\{1,\ldots,n\}$ having probability $\pi(i)$ probability of $X = i$. Then by Jensen's inequality and the convexity of $f(x) = -\log(x)$, $$ G = 2N \mathbb E\left[-\log\frac{\gamma(X)}{\pi(X)}\right] \ge -2N \log \mathbb E\frac{\gamma(X)}{\pi(X)}. $$ But $$ \mathbb E \frac{\gamma(X)}{\pi(X)} = \sum_i \pi(i) \frac{\gamma(i)}{\pi(i)} = \sum_i \gamma(i) = 1, $$ so $G \ge -2N \log(1) = 0$.

$\endgroup$
1
  • $\begingroup$ +1. However, in your first sentence you mean non-negative rather than positive -- your demonstration gives that it is $\geq 0$ and the bound is attainable; $G$ can actually be zero. For an example, see the initial case in the example in my answer which had $\pi = (0.6,0.4)^\prime$ and $O=(6,4)^\prime$), so $G=0$. $\endgroup$
    – Glen_b
    Oct 16, 2018 at 22:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.