# Chi-square goodness of fit with few categories

I am trying to compare observed proportions to expected proportions. To me, they seem quite different but chisq.test stubbornly returns fairly high p-values. I think the problem might be that I have few categories:

a <- c(10,30,50,70)
b <- c(70,50,30,10)
chisq.test (a, b)
...
p-value = 0.2133


but when I add a few categories which, to my eyes, make the two sets actually more similar, the p-value drops:

a <- c(10,30,50,70,20,20,20,20)
b <- c(70,50,30,10,20,20,20,20)
chisq.test (a, b)
...
p-value = 0.01


I understand I am missing something about this test, how it is supposed to be used, or which test I ought to be using in this case, but I don't seem to be able to figure out which it is or how to fix it.

• Try chisq.test(rbind(a,b)) and you then get the same $\chi^2$ statistic of $100$ but more degrees of freedom in your second example so you should get a higher $p$ value. They are still very small: $1.6 \times 10^{-21}$ in your first example and $1.1 \times 10^{-18}$ in your second, so you may not see a difference May 4, 2019 at 20:43
• Generally a chisq.test in R would perform contingency table tests. If you want to want to perform a goodness of fit test, try converting one of the vectors to probabilities and pass it as a probability argument to the function. Read more here May 5, 2019 at 1:09
• @Henry Thanks, but I think I wasn't clear, sorry. a are supposed to be my observed frequencies, and b my expected ones. Your solution will result in a and b combined being compared against a uniform distribution. May 5, 2019 at 7:14
• @SanketAgrawal Thanks. You nailed it, but it's only now that I understood your answer, after I had understood my mistake. May 5, 2019 at 7:15

As suggested by @Henry, I find that the best way to do such a chi-squared test in R is to use a matrix containing the observed counts.

MAT = cbind(a,b); MAT
a  b
[1,] 10 70
[2,] 30 50
[3,] 50 30
[4,] 70 10

chisq.test(MAT)

Pearson's Chi-squared test

data:  MAT
X-squared = 100, df = 3, p-value < 2.2e-16


So you have a highly significant result, as you say you expected from these data.

Whenever you use software to do a new procedure, it is best to check the program--and your syntax using the software--with a worked example. In this case, the chi-squared statistic is $$Q = \sum_{i=1}^4 \sum_{j=1}^2 \frac{(X_{ij} - E_{ij})^2}{E_{ij}},$$

where $$X_{ij}$$ are the observed counts, $$E_{ij}$$ are expected counts found according to the null hypothesis, and $$Q \stackrel{aprx}{\sim}\mathsf{Chisq}(df = 3),$$ provided the $$E_{ij}$$ are sufficiently large. (Many authors say it is best if all $$E_{ij} \ge 5.)$$

[The formula for the number of degrees of freedom is $$(r-1)(c-1) = (4-1)(2-1) = 3,$$ where $$r$$ and $$c$$ are the respective numbers of rows and columns in the matrix.]

There is more information lurking behind the scenes in R's chisq.test procedure than is immediately displayed. Here is some of that additional information.

First, you can 'echo back' the observed counts, to make sure they are as you intended. Note the use of $-notation to access additional information. chisq.out = chisq.test(MAT) chisq.out$obs
a  b
[1,] 10 70
[2,] 30 50
[3,] 50 30
[4,] 70 10


So $$X_{11} = 10.$$ Now for the expected counts.

chisq.out$exp a b [1,] 40 40 [2,] 40 40 [3,] 40 40 [4,] 40 40  In this case, it turns out that the expected counts are all the same, but that is not always the case. Please look at your text or class notes to see how the $$E_{ij}$$ are computed for this kind of test. Here we have $$E_{11} = 40.$$ According the the formula displayed above there are $$4 \times 2 = 8$$ contributions $$C_{ij} =(X_{ij} - E_{ij})^2/E_{ij}$$ to the chi-squared statistic $$Q.$$ In particular, contribution $$C_{11} = (10 - 40)^2/40 = 22.5.$$ It is also a good idea to look at the 'Pearson residuals'. They are $$R_{ij} = \pm \sqrt{C_{ij}},$$ where the sign is determined by whether $$X_{ij} - E_{ij}$$ is positive or negative. So $$R_{11} = -4.74,$$ which agrees with the output below. chisq.test(MAT)$resid
a         b
[1,] -4.743416  4.743416
[2,] -1.581139  1.581139
[3,]  1.581139 -1.581139
[4,]  4.743416 -4.743416


Many textbooks invite you to focus attention on any Pearson residuals with absolute values above about 2. Those indicate cells in the table that may have contributed to a significant result.

In your case, it seems that the first and last row categories may not be 'behaving' according to the null hypothesis.

Note: You are, by far, not the first person to be confused by the syntax for the R function chisq.test. While the documentation is technically correct, IMHO it should be reworded to avoid unnecessary confusion.

Here is a situation in which a and b are 'factor' variables, as required for the syntax you used. A fair die is rolled 600 times and the results recorded in a. A 'psychic', isolated in another, room tries to guess each result in sequence. Her results are recorded in b. Are a and b independent $$(H_0),$$ or not? An analysis of a simulated experiment (modeled with no psychic ability) is as follows:

a = as.factor(sample(1:6, 600, rep=T))
b = as.factor(sample(1:6, 600, rep=T))
chi.out = chisq.test(a,b)
chi.out\$obs
b
a    1  2  3  4  5  6
1 20 11 20 15 15 15
2 13 18 22 11 14 16
3 16 21 14 17 20 20
4 16  8 10 16 14 24
5 17 14 20 23 23 13
6 19 18 19 19 15 14

chi.out

Pearson's Chi-squared test

data:  a and b
X-squared = 26.957, df = 25, p-value = 0.358

• Yes, thank you, this is it! I should have done chisq.test (a, p=b/sum(b)). But thanks for the broader explanation, I appreciate it, and it's been really helpful. May 5, 2019 at 7:17
• Generally speaking chi-sq tests of independence and homogeneity all "compare observed proportions to expected proportions." In your problem it would have been clearer to specify that a contains observed counts and b expected counts in the same 4 levels of one categorical variable--if that is the case. Then your code is OK. May 5, 2019 at 17:22