Why does independence test use the chi-squared distribution?

The $\chi^2$ goodness-of-fit test uses the following statistic: $$\chi_0^2=\sum_{i=1}^n\frac{(O_i-E_i)^2}{E_i}$$ In the test, granting that the conditions are met, one uses the $\chi^2$-distribution to calculate the p-value that given the $H_0$ is true one would observe such a value in a representative sample of the same size.

However, in order for a statistic $\chi_0^2$ to follow a $\chi^2$-distribution (with $n-1$ degrees of freedom), it must be true that: $$\sum_{i=1}^n\frac{(O_i-E_i)^2}{E_i}=\sum_{i=1}^{n-1}Z_i^2$$ for independent, standard normal $Z_i$ (Wikipedia). The conditions for the test are as follows (again, from Wikipedia):

1. Sample representative of population
2. Large sample size
3. Expected cell count is sufficiently large
4. Independence between each category

From conditions (1,2) it is clear that we satisfy conditions for inference from the sample to the population. (3) seems to be an assumption required because the discrete count $E_i$, which is in the denominator, does not result in a near-continuous distribution for each $Z_i$ and if it is not large enough there is an error that can be corrected with Yates' correction - this seems to be from the fact that a discrete distribution is basically a "floored" continuous one, so the shift by $1/2$ for each one corrects this.

The necessity of (4) seems to come in handy later, but I cannot see how.

At first, I thought that $Z_i=\frac{O_i-E_i}{\sqrt{E_i}}$ is necessary for the statistic to match the distribution. This lead me to the questionable assumption that $O_i-E_i\sim \mathcal{N}(0, \sqrt{E_i})$, which was indeed wrong. In fact, it is clear from the reduction of dimension for two sides of the equality from $n$ to $n-1$ that this cannot be the case.

It has become apparent, thanks to whuber's explanations, that $Z_i$ need not equal each $\frac{O_i-E_i}{\sqrt{E_i}}$ term because $\chi_0^2=\sum_{i=1}^{n-1}Z_i^2$ (note the reduction in the number of summed variables) for standard normal random variables $Z_i$ which are functionally independent.

My question, then, is how can $\chi_0^2$ follow the $\chi^2$ distribution? What kinds of combinations of each of the $\frac{(O_i-E_i)^2}{E_i}$ terms result in squared standard normals $Z_i^2$? This requires the use of the CLT, apparently (and that makes sense), but how? In other words, what is each $Z_i$ equal to (or approximately equal to)?

• I am curious where you read that anybody assumes the last thing that you stated ($O_i-E_i\sim \mathcal{N}(0, \sqrt{E_i})$). That is not necessary: the $\chi^2$ statistic can have a $\chi^2$ distribution (at least to an extremely good approximation) without any of these standardized residuals having a normal distribution. The question you seem to want to ask is how do these assumptions justify referring the $\chi^2$ statistic to a $\chi^2$ distribution? By themselves, they do not. For a discussion of what can go wrong, please see my post at stats.stackexchange.com/a/17148.
– whuber
Jan 14, 2014 at 16:19
• From the equality of two sums of squares you cannot conclude the square roots are equal term by term! Because that is the case for mere numbers, it surely is the case for random variables, too.
– whuber
Jan 14, 2014 at 16:28
• To make this concrete, suppose $(W_i),i=1, \ldots,n$ are independently distributed with $\chi$ distributions having degrees of freedom $\nu_1,\nu_2,\ldots,\nu_n$ and that $\nu_1+\nu_2+\cdots+\nu_n=n-1$ but $\nu_i\ne 1$ for all $i$. Then although none of the $W_i$ is normal, nevertheless $\sum_{i=1}^n W_i^2$ has a $\chi^2(n-1)$ distribution.
– whuber
Jan 14, 2014 at 16:37
• If by "squared standard normal" you mean "sum of independent squared standard normals," that's the question I believe you really wanted to pose at the outset :-). And in the end, most analyses of the situation do indeed invoke the Central Limit Theorem to prove that the standardized residuals asymptotically are standard normal (but not quite independent, which is why the degrees of freedom are $n-1$ and not $n$).
– whuber
Jan 14, 2014 at 16:58
• +1 for what I anticipate will soon be a very good question. The first problem is the independence test doesn't use the claimed statistic. The statistic given at the start is unidimensional (a sum over $n$ categories), while a test of independence requires more than one variable. Please edit to make the name of the test and the statistic correspond. Jan 14, 2014 at 21:44

2 Answers

It's about the Poisson distribution. If $X$ is Poisson with mean $\lambda$, then the variance of $X$ is $\lambda$ also. This means that $$\frac{(X-\lambda)^2}{\lambda}$$ is a $z^2$ like entity. By the CLT, the Poisson tends to normality as the mean gets large, which is where the chi-squared comes in. Yes, it is an asymptotic test.

The degrees of freedom come from Cochran's theorem. Basically, Cochran explains how the Chi-squared is transformed (or remains unchanged) subject to a linear transformation in the $z^2$ scores.

$$\sum_i z_i^2=Z' I Z$$

in matrix notation. If instead of computing the usual sum of squares, you compute $$Z' Q Z$$ for some matrix Q, then you still get a quantity with a a chi-squared distribution, but the degrees of freedom are now the rank of $Q$. There are more conditions on the matrix Q, but this is the gist of it.

If you play around with some matrix notation, you can express $$\sum_i (z_i-\bar{z})^2$$ as a quadratic form. Cochran assumes independence of the original normal variates, which is why the columns of your table of counts must be independent as well.

• Sorry, but you've definitely lost me at "If instead, you do..."
– VF1
Jan 15, 2014 at 3:13
• @VF1, I made a change, so I hope it's more clear. Cochrane's theorem is the answer to your question of when a sum of squares with normals in it has a chi-squared distribution. Jan 15, 2014 at 3:29
• OK, I will take a look at this. I'll leave the question open, though, in case anyone else has something to add.
– VF1
Jan 15, 2014 at 3:33
• Ordinarily the sample size is fixed. That means it's impossible that any of the entries could follow a Poisson distribution. The appeal to a Poisson distribution therefore looks like it's just another approximation--and seems to leave us right where we started.
– whuber
Jul 17, 2016 at 19:16

According to the textbook "Introductory Statistics with Randomization and Simulation", section 3.3.2 (textbook freely available at OpenIntro), the $\chi^2$ test statistic is trying to accumulate the deviations of the observed from the expected. And the deviations are indeed expressed through the term

$$Z_i = \frac{O_i - E_i}{\sqrt{E_i}}$$

which actually originates from $$\frac{O_i - E_i}{(Standard Error Of The Observed)}$$.

The textbook goes on to say that the $(StandardErrorOfTheObserved)$ is better estimated by $\sqrt{E_i}$, so the term becomes $Z_i = \frac{O_i - E_i}{\sqrt{E_i}}$. The textbook doesn't actually explain why this substitution is acceptable, and I'd also like to find out.

Anyway, you could create a test statistic of the form

$$Z = |Z_1| + |Z_2| + |Z_3| + ...$$

but it's better to square all the terms, because you get positive values immediately and the higher values stand out more after squaring. So you get the following: $$\chi^2 = Z_1^2 + Z_2^2 + Z_3^2 +...$$

But I don't know either why should this sum follow the $\chi^2$ distribution, or what's the connection to the definition of the $\chi^2$ distribution (sum of squares of standard normal independent variables).

EDIT: I'm still learning statistics, and I still don't think I understand the $\chi^2$ test properly. I hope others can enlighten me too.