7
$\begingroup$

I have read previous questions on this site and don't quite understand fully and hopefully my question is sufficiently different. I'm certain to have misconceptions and would be horrified in passing any on to my students. I'm going to be teaching $\chi^2$ tests with contingency tables and would like to find out more about why the test works. Many sources simply detail the mechanics of the test and I want to understand more than that. To give you an idea of the level of detail I'm after here is my current understanding (please correct any errors)

The counts in each cell are from a multinomial distribution $\{n_i\} \sim mult(N,\{p_i\})$

The counts in a particular cell are distributed $n_i\sim bin(N,p_i)$

We approximate the binomial distribution with a poisson with $\mu =E(x)$ and $\sigma^2=E(x)$

We form the test statistic $X^2=\sum \frac{(O-E)^2}{E}$ which is $\chi^2$ like.

If $\mu=E(x) $ is large enough in a cell then $\frac{(O-E)^2}{E}$ is approximately $z^2$ hence the greater than 5 in each cell.

The degrees of freedom is $\nu=(r-1)(c-1)$ taking into account the cells that'd be fixed.

Compare with $\chi^2_{\nu}=Z^2_1+\cdots +Z^2_{\nu}$

Reconciling the two ways of using degrees of freedom is particularly difficult for me. The best book I have is 'Understanding Statistics' and I want the next level above that I think.

Any help would be greatly appreciated.

$\endgroup$
  • 1
    $\begingroup$ You can find nice introduction in Fisher's (1925) text: psychclassics.yorku.ca/Fisher/Methods/chap4.htm $\endgroup$ – Tim Jan 23 '15 at 11:02
  • $\begingroup$ Just looking at it now thanks. Is $S$ the notation for $\Sigma$? $\endgroup$ – Karl Jan 23 '15 at 19:13
  • $\begingroup$ Yes, strangely it is. $\endgroup$ – Tim Jan 23 '15 at 19:22
3
$\begingroup$

From a memorable, intuitive perspective, your account is fine. The considerations of degrees of freedom rest on the understanding that each standardized residual,

$$Z_i = \frac{O_i-E_i}{\sqrt{E_i}},$$

is close enough to having a standard Normal distribution that the sum of their squares

$$X^2 = \sum Z_i^2 = \sum_i^n \frac{(O_i-E_i)^2}{E_i}$$

has a distribution close to that of the sum of $n$ standard Normal variables. (For convenience I have written $n=rc$ and index the residuals from $i=1$ to $i=rc$, rather than following the double-indexing that is usually used in two-way tables.)

The problem, as every textbook at every level points out, is that the $Z_i$ are not independent. In fact, they satisfy a lot of linear relations. Across each row, for instance, the sum of the $O_i$ (counts of observations) equals the sum of the $E_i$ (their expectations, constructed to give the same sum), whence the sum of the $Z_i$ across each row is zero. Similar, the column sums of the $Z_i$ are zero. But many more sums than that are zero: any linear combination of such sums will also be zero.

The best solution is to consider this geometrically. The vector $\mathbf{Z}=(Z_1, \ldots, Z_n)$ can be located anywhere in $\mathbb{R}^n$. (Visualize this in $n=3$ dimensions.) If the $Z_i$ truly were independent, they would have a rotationally-invariant distribution around the origin. The only variation in their probability density would be radial. Indeed, $X^2$ is precisely the squared length of $\mathbf{Z}$.

Any single sum-to-zero constraint defines a hyperplane in this space. (Visualize a plane in $\mathbb{R}^3$, such as the $xy$ plane.) The constraint restricts the vector $\mathbf{Z}$ to lie within that hyperplane. Nevertheless, because any rotation in that hyperplane can be extended to a rotation of the entire space--just fix the perpendicular direction--the distribution of $\mathbf{Z}$ remains rotationally invariant in this hyperplane. It therefore is identical to the distribution of independent standard Normal variables within the hyperplane itself, which has one less dimension, $n-1$.

(In the running example, $\mathbf{Z}=(Z_1,Z_2,Z_3)$ when restricted to the $xy$ plane in $\mathbb{R}^3$ is just $(Z_1,Z_2)$. Therefore its squared length is just $Z_1^2 + Z_2^2$, which by definition has a $\chi^2(2)$ distribution. Due to the rotational invariance of $\mathbf{Z}$, the distribution of $|\mathbf{Z}|^2$ in any plane through the origin in $\mathbb{R}^3$ will be the same as this one: namely, $\chi^2(2)$, not $\chi^2(3)$.)

In this fashion each additional hyperplane thereby reduces the number of independent standard Normal variables in the description, provided it truly reduces the dimension of the space in which $\mathbf{Z}$ may lie. The actual mathematical content of the $\nu = (r-1)(c-1)$ equation is to assert that (a) the $r+c$ obvious sum-to-zero constraints in the $r\times c$ table amount only to $r+c-1$ independent constraints--there is one degree of redundancy--and (b) there are no other constraints independent of them. Proving these facts is a matter of linear algebra: that is, of solving systems of equations and counting dimensions.

(It is easy to see there are at most $r+c-1$ independent constraints, because the fact that all $n$ $Z_i$ sum to zero follows by adding either all the row sums or all the column sums, showing there is at least this degree of redundancy among the row-sum and column-sum constraints.)

Consequently, we should visualize the situation as one of looking at the distribution of $rc$ independent standard Normal variables within a linear subspace that is determined by the intersection of $r + c - 1 $ independent hyperplanes. Within that subspace, the sum of squares of those variables still gives the distance to the origin. That distance, however, lies within a space of dimension only

$$\nu = rc - (r + c - 1) = (r-1)(c-1).$$

Thus, the distribution of the sum of squares is the same as the distribution of the sum of squares of $(r-1)(c-1)$ independent standard Normal variables.


The answers at How to understand degrees of freedom? provide additional explanations, many from similar viewpoints but at varying levels of sophistication and rigor.

$\endgroup$
  • $\begingroup$ How beautiful would a 3D plot be to illustrate this instant classic... $\endgroup$ – Antoni Parellada Jul 4 '16 at 21:04
  • $\begingroup$ @Antoni Such plots already exist, both in the thread I referenced and in several other related ones. $\endgroup$ – whuber Jul 4 '16 at 21:05
  • 1
    $\begingroup$ I like the plot at the end of the hyperlinked thread... Completely forgot about that one... LOL $\endgroup$ – Antoni Parellada Jul 4 '16 at 21:06
  • 1
    $\begingroup$ While looking at these issues geometrically is often a really good way to approach them -- often rendering otherwise seemingly tricky ideas obvious -- I find that many students these days seem to lack any geometric intuition at all; when giving geometric arguments/motivations for things I try to have several other ways of approach it for the many students that lack it. $\endgroup$ – Glen_b Jul 5 '16 at 1:00
  • 1
    $\begingroup$ I have had some bizarre conversations with highly intelligent people that simply refused to accept what seemed to me obvious intuitions (such as one person that held false ideas relating to probability but refused to reason along the lines that if they were correct they ought to be prepared to consider what would happen if they bet according to their intuition -- the entire line of thought-experiment was simply "out of bounds". Not on any religious or moral grounds, it was simply an intellectual step they refused to contemplate. It can be hard to achieve much in the face of that kind of thing) $\endgroup$ – Glen_b Jul 5 '16 at 1:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.