12
$\begingroup$

The $\chi^2$ goodness-of-fit test uses the following statistic: $$ \chi_0^2=\sum_{i=1}^n\frac{(O_i-E_i)^2}{E_i} $$ In the test, granting that the conditions are met, one uses the $\chi^2$-distribution to calculate the p-value that given the $H_0$ is true one would observe such a value in a representative sample of the same size.

However, in order for a statistic $\chi_0^2$ to follow a $\chi^2$-distribution (with $n-1$ degrees of freedom), it must be true that: $$ \sum_{i=1}^n\frac{(O_i-E_i)^2}{E_i}=\sum_{i=1}^{n-1}Z_i^2 $$ for independent, standard normal $Z_i$ (Wikipedia). The conditions for the test are as follows (again, from Wikipedia):

  1. Sample representative of population
  2. Large sample size
  3. Expected cell count is sufficiently large
  4. Independence between each category

From conditions (1,2) it is clear that we satisfy conditions for inference from the sample to the population. (3) seems to be an assumption required because the discrete count $E_i$, which is in the denominator, does not result in a near-continuous distribution for each $Z_i$ and if it is not large enough there is an error that can be corrected with Yates' correction - this seems to be from the fact that a discrete distribution is basically a "floored" continuous one, so the shift by $1/2$ for each one corrects this.

The necessity of (4) seems to come in handy later, but I cannot see how.

At first, I thought that $Z_i=\frac{O_i-E_i}{\sqrt{E_i}}$ is necessary for the statistic to match the distribution. This lead me to the questionable assumption that $O_i-E_i\sim \mathcal{N}(0, \sqrt{E_i})$, which was indeed wrong. In fact, it is clear from the reduction of dimension for two sides of the equality from $n$ to $n-1$ that this cannot be the case.

It has become apparent, thanks to whuber's explanations, that $Z_i$ need not equal each $\frac{O_i-E_i}{\sqrt{E_i}}$ term because $\chi_0^2=\sum_{i=1}^{n-1}Z_i^2$ (note the reduction in the number of summed variables) for standard normal random variables $Z_i$ which are functionally independent.

My question, then, is how can $\chi_0^2$ follow the $\chi^2$ distribution? What kinds of combinations of each of the $\frac{(O_i-E_i)^2}{E_i}$ terms result in squared standard normals $Z_i^2$? This requires the use of the CLT, apparently (and that makes sense), but how? In other words, what is each $Z_i$ equal to (or approximately equal to)?

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ I am curious where you read that anybody assumes the last thing that you stated ($O_i-E_i\sim \mathcal{N}(0, \sqrt{E_i})$). That is not necessary: the $\chi^2$ statistic can have a $\chi^2$ distribution (at least to an extremely good approximation) without any of these standardized residuals having a normal distribution. The question you seem to want to ask is how do these assumptions justify referring the $\chi^2$ statistic to a $\chi^2$ distribution? By themselves, they do not. For a discussion of what can go wrong, please see my post at stats.stackexchange.com/a/17148. $\endgroup$ – whuber♦ Jan 14 '14 at 16:19
  • 1
    $\begingroup$ From the equality of two sums of squares you cannot conclude the square roots are equal term by term! Because that is the case for mere numbers, it surely is the case for random variables, too. $\endgroup$ – whuber♦ Jan 14 '14 at 16:28
  • 1
    $\begingroup$ To make this concrete, suppose $(W_i),i=1, \ldots,n$ are independently distributed with $\chi$ distributions having degrees of freedom $\nu_1,\nu_2,\ldots,\nu_n$ and that $\nu_1+\nu_2+\cdots+\nu_n=n-1$ but $\nu_i\ne 1$ for all $i$. Then although none of the $W_i$ is normal, nevertheless $\sum_{i=1}^n W_i^2$ has a $\chi^2(n-1)$ distribution. $\endgroup$ – whuber♦ Jan 14 '14 at 16:37
  • 1
    $\begingroup$ If by "squared standard normal" you mean "sum of independent squared standard normals," that's the question I believe you really wanted to pose at the outset :-). And in the end, most analyses of the situation do indeed invoke the Central Limit Theorem to prove that the standardized residuals asymptotically are standard normal (but not quite independent, which is why the degrees of freedom are $n-1$ and not $n$). $\endgroup$ – whuber♦ Jan 14 '14 at 16:58
  • 1
    $\begingroup$ +1 for what I anticipate will soon be a very good question. The first problem is the independence test doesn't use the claimed statistic. The statistic given at the start is unidimensional (a sum over $n$ categories), while a test of independence requires more than one variable. Please edit to make the name of the test and the statistic correspond. $\endgroup$ – Glen_b♦ Jan 14 '14 at 21:44
6
$\begingroup$

It's about the Poisson distribution. If $X$ is Poisson with mean $\lambda$, then the variance of $X$ is $\lambda$ also. This means that $$\frac{(X-\lambda)^2}{\lambda}$$ is a $z^2$ like entity. By the CLT, the Poisson tends to normality as the mean gets large, which is where the chi-squared comes in. Yes, it is an asymptotic test.

The degrees of freedom come from Cochran's theorem. Basically, Cochran explains how the Chi-squared is transformed (or remains unchanged) subject to a linear transformation in the $z^2$ scores.

$$\sum_i z_i^2=Z' I Z$$

in matrix notation. If instead of computing the usual sum of squares, you compute $$Z' Q Z$$ for some matrix Q, then you still get a quantity with a a chi-squared distribution, but the degrees of freedom are now the rank of $Q$. There are more conditions on the matrix Q, but this is the gist of it.

If you play around with some matrix notation, you can express $$\sum_i (z_i-\bar{z})^2$$ as a quadratic form. Cochran assumes independence of the original normal variates, which is why the columns of your table of counts must be independent as well.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Sorry, but you've definitely lost me at "If instead, you do..." $\endgroup$ – VF1 Jan 15 '14 at 3:13
  • $\begingroup$ @VF1, I made a change, so I hope it's more clear. Cochrane's theorem is the answer to your question of when a sum of squares with normals in it has a chi-squared distribution. $\endgroup$ – Placidia Jan 15 '14 at 3:29
  • 1
    $\begingroup$ OK, I will take a look at this. I'll leave the question open, though, in case anyone else has something to add. $\endgroup$ – VF1 Jan 15 '14 at 3:33
  • 1
    $\begingroup$ Ordinarily the sample size is fixed. That means it's impossible that any of the entries could follow a Poisson distribution. The appeal to a Poisson distribution therefore looks like it's just another approximation--and seems to leave us right where we started. $\endgroup$ – whuber♦ Jul 17 '16 at 19:16
1
$\begingroup$

According to the textbook "Introductory Statistics with Randomization and Simulation", section 3.3.2 (textbook freely available at OpenIntro), the $\chi^2$ test statistic is trying to accumulate the deviations of the observed from the expected. And the deviations are indeed expressed through the term

$$Z_i = \frac{O_i - E_i}{\sqrt{E_i}}$$

which actually originates from $$\frac{O_i - E_i}{(Standard Error Of The Observed)}$$.

The textbook goes on to say that the $(StandardErrorOfTheObserved)$ is better estimated by $\sqrt{E_i}$, so the term becomes $Z_i = \frac{O_i - E_i}{\sqrt{E_i}}$. The textbook doesn't actually explain why this substitution is acceptable, and I'd also like to find out.

Anyway, you could create a test statistic of the form

$$Z = |Z_1| + |Z_2| + |Z_3| + ...$$

but it's better to square all the terms, because you get positive values immediately and the higher values stand out more after squaring. So you get the following: $$\chi^2 = Z_1^2 + Z_2^2 + Z_3^2 +...$$

But I don't know either why should this sum follow the $\chi^2$ distribution, or what's the connection to the definition of the $\chi^2$ distribution (sum of squares of standard normal independent variables).

EDIT: I'm still learning statistics, and I still don't think I understand the $\chi^2$ test properly. I hope others can enlighten me too.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.