Why does independence test use the chi-squared distribution?

Question

The $\chi^2$ goodness-of-fit test uses the following statistic: $$ \chi_0^2=\sum_{i=1}^n\frac{(O_i-E_i)^2}{E_i} $$ In the test, granting that the conditions are met, one uses the $\chi^2$-distribution to calculate the p-value that given the $H_0$ is true one would observe such a value in a representative sample of the same size.

However, in order for a statistic $\chi_0^2$ to follow a $\chi^2$-distribution (with $n-1$ degrees of freedom), it must be true that: $$ \sum_{i=1}^n\frac{(O_i-E_i)^2}{E_i}=\sum_{i=1}^{n-1}Z_i^2 $$ for independent, standard normal $Z_i$ (Wikipedia). The conditions for the test are as follows (again, from Wikipedia):

1. Sample representative of population
2. Large sample size
3. Expected cell count is sufficiently large
4. Independence between each category

From conditions (1,2) it is clear that we satisfy conditions for inference from the sample to the population. (3) seems to be an assumption required because the discrete count $E_i$, which is in the denominator, does not result in a near-continuous distribution for each $Z_i$ and if it is not large enough there is an error that can be corrected with Yates' correction - this seems to be from the fact that a discrete distribution is basically a "floored" continuous one, so the shift by $1/2$ for each one corrects this.

The necessity of (4) seems to come in handy later, but I cannot see how.

At first, I thought that $Z_i=\frac{O_i-E_i}{\sqrt{E_i}}$ is necessary for the statistic to match the distribution. This lead me to the questionable assumption that $O_i-E_i\sim \mathcal{N}(0, \sqrt{E_i})$, which was indeed wrong. In fact, it is clear from the reduction of dimension for two sides of the equality from $n$ to $n-1$ that this cannot be the case.

It has become apparent, thanks to whuber's explanations, that $Z_i$ need not equal each $\frac{O_i-E_i}{\sqrt{E_i}}$ term because $\chi_0^2=\sum_{i=1}^{n-1}Z_i^2$ (note the reduction in the number of summed variables) for standard normal random variables $Z_i$ which are functionally independent.

My question, then, is how can $\chi_0^2$ follow the $\chi^2$ distribution? What kinds of combinations of each of the $\frac{(O_i-E_i)^2}{E_i}$ terms result in squared standard normals $Z_i^2$? This requires the use of the CLT, apparently (and that makes sense), but how? In other words, what is each $Z_i$ equal to (or approximately equal to)?

Answer 1

It's about the Poisson distribution. If $X$ is Poisson with mean $\lambda$, then the variance of $X$ is $\lambda$ also. This means that $$\frac{(X-\lambda)^2}{\lambda}$$ is a $z^2$ like entity. By the CLT, the Poisson tends to normality as the mean gets large, which is where the chi-squared comes in. Yes, it is an asymptotic test.

The degrees of freedom come from Cochran's theorem. Basically, Cochran explains how the Chi-squared is transformed (or remains unchanged) subject to a linear transformation in the $z^2$ scores.

$$\sum_i z_i^2=Z' I Z$$

in matrix notation. If instead of computing the usual sum of squares, you compute $$Z' Q Z$$ for some matrix Q, then you still get a quantity with a a chi-squared distribution, but the degrees of freedom are now the rank of $Q$. There are more conditions on the matrix Q, but this is the gist of it.

If you play around with some matrix notation, you can express $$\sum_i (z_i-\bar{z})^2$$ as a quadratic form. Cochran assumes independence of the original normal variates, which is why the columns of your table of counts must be independent as well.

Answer 2

According to the textbook "Introductory Statistics with Randomization and Simulation", section 3.3.2 (textbook freely available at OpenIntro), the $\chi^2$ test statistic is trying to accumulate the deviations of the observed from the expected. And the deviations are indeed expressed through the term

$$Z_i = \frac{O_i - E_i}{\sqrt{E_i}}$$

which actually originates from $$\frac{O_i - E_i}{(Standard Error Of The Observed)}$$.

The textbook goes on to say that the $(StandardErrorOfTheObserved)$ is better estimated by $\sqrt{E_i}$, so the term becomes $Z_i = \frac{O_i - E_i}{\sqrt{E_i}}$. The textbook doesn't actually explain why this substitution is acceptable, and I'd also like to find out.

Anyway, you could create a test statistic of the form

$$Z = |Z_1| + |Z_2| + |Z_3| + ...$$

but it's better to square all the terms, because you get positive values immediately and the higher values stand out more after squaring. So you get the following: $$\chi^2 = Z_1^2 + Z_2^2 + Z_3^2 +...$$

But I don't know either why should this sum follow the $\chi^2$ distribution, or what's the connection to the definition of the $\chi^2$ distribution (sum of squares of standard normal independent variables).

EDIT: I'm still learning statistics, and I still don't think I understand the $\chi^2$ test properly. I hope others can enlighten me too.