Chi-square goodness of fit with few categories

Question

I am trying to compare observed proportions to expected proportions. To me, they seem quite different but chisq.test stubbornly returns fairly high p-values. I think the problem might be that I have few categories:

a <- c(10,30,50,70)
b <- c(70,50,30,10)
chisq.test (a, b)
...
p-value = 0.2133

but when I add a few categories which, to my eyes, make the two sets actually more similar, the p-value drops:

a <- c(10,30,50,70,20,20,20,20)
b <- c(70,50,30,10,20,20,20,20)
chisq.test (a, b)
...
p-value = 0.01

I understand I am missing something about this test, how it is supposed to be used, or which test I ought to be using in this case, but I don't seem to be able to figure out which it is or how to fix it.

Answer 1

As suggested by @Henry, I find that the best way to do such a chi-squared test in R is to use a matrix containing the observed counts.

MAT = cbind(a,b); MAT
      a  b
[1,] 10 70
[2,] 30 50
[3,] 50 30
[4,] 70 10

chisq.test(MAT)

        Pearson's Chi-squared test

data:  MAT
X-squared = 100, df = 3, p-value < 2.2e-16

So you have a highly significant result, as you say you expected from these data.

Whenever you use software to do a new procedure, it is best to check the program--and your syntax using the software--with a worked example. In this case, the chi-squared statistic is $$Q = \sum_{i=1}^4 \sum_{j=1}^2 \frac{(X_{ij} - E_{ij})^2}{E_{ij}},$$

where $X_{ij}$ are the observed counts, $E_{ij}$ are expected counts found according to the null hypothesis, and $Q \stackrel{aprx}{\sim}\mathsf{Chisq}(df = 3),$ provided the $E_{ij}$ are sufficiently large. (Many authors say it is best if all $E_{ij} \ge 5.)$

[The formula for the number of degrees of freedom is $(r-1)(c-1) = (4-1)(2-1) = 3,$ where $r$ and $c$ are the respective numbers of rows and columns in the matrix.]

There is more information lurking behind the scenes in R's chisq.test procedure than is immediately displayed. Here is some of that additional information.

First, you can 'echo back' the observed counts, to make sure they are as you intended. Note the use of $-notation to access additional information.

chisq.out = chisq.test(MAT)
chisq.out$obs
      a  b
[1,] 10 70
[2,] 30 50
[3,] 50 30
[4,] 70 10

So $X_{11} = 10.$ Now for the expected counts.

chisq.out$exp
      a  b
[1,] 40 40
[2,] 40 40
[3,] 40 40
[4,] 40 40

In this case, it turns out that the expected counts are all the same, but that is not always the case. Please look at your text or class notes to see how the $E_{ij}$ are computed for this kind of test. Here we have $E_{11} = 40.$

According the the formula displayed above there are $ 4 \times 2 = 8$ contributions $C_{ij} =(X_{ij} - E_{ij})^2/E_{ij}$ to the chi-squared statistic $Q.$ In particular, contribution $C_{11} = (10 - 40)^2/40 = 22.5.$

It is also a good idea to look at the 'Pearson residuals'. They are $R_{ij} = \pm \sqrt{C_{ij}},$ where the sign is determined by whether $X_{ij} - E_{ij}$ is positive or negative. So $R_{11} = -4.74,$ which agrees with the output below.

chisq.test(MAT)$resid
             a         b
[1,] -4.743416  4.743416
[2,] -1.581139  1.581139
[3,]  1.581139 -1.581139
[4,]  4.743416 -4.743416

Many textbooks invite you to focus attention on any Pearson residuals with absolute values above about 2. Those indicate cells in the table that may have contributed to a significant result.

In your case, it seems that the first and last row categories may not be 'behaving' according to the null hypothesis.

Note: You are, by far, not the first person to be confused by the syntax for the R function chisq.test. While the documentation is technically correct, IMHO it should be reworded to avoid unnecessary confusion.

Here is a situation in which a and b are 'factor' variables, as required for the syntax you used. A fair die is rolled 600 times and the results recorded in a. A 'psychic', isolated in another, room tries to guess each result in sequence. Her results are recorded in b. Are a and b independent $(H_0),$ or not? An analysis of a simulated experiment (modeled with no psychic ability) is as follows:

a = as.factor(sample(1:6, 600, rep=T))
b = as.factor(sample(1:6, 600, rep=T))
chi.out = chisq.test(a,b)
chi.out$obs  
   b
a    1  2  3  4  5  6
  1 20 11 20 15 15 15
  2 13 18 22 11 14 16
  3 16 21 14 17 20 20
  4 16  8 10 16 14 24
  5 17 14 20 23 23 13
  6 19 18 19 19 15 14

chi.out

        Pearson's Chi-squared test

data:  a and b
X-squared = 26.957, df = 25, p-value = 0.358