# How do I reject or fail to reject the null hypothesis that p1 = p2 = p3 = p4?

I need to do reliability analysis on computerized test questions. Given a multiple choice question, if 20/25 got it right the first time the test was given, 22/30 the second time the test was given, 15/22 the third time the test was given, and 14/23 the fourth time the test was given, I want to determine if all of these came from the same population or not. [I assume the population is the same, but is the question reliable?] If I fail to reject the null hypothesis the assumption is reliability, although technically not proven.

I am thinking of using ANOVA on this for proportions, but I don't know that the necessary assumptions have been met to do this. My sample sizes are small. What do you recommend?

-

You have a contingency table with $C=2$ columns and $R=4$ rows.

       Right    Wrong
1st     20        5       25
2nd     22        8       30
3rd     15        7       22
4th     14        9       23
71       29      100


Let $N_{ij}$ be the number of observations in the $(i,j)$-cell. Define $$N_{i\bullet}=\sum_{j=1}^C N_{ij} \qquad \textrm{and} \qquad N_{\bullet j}=\sum_{i=1}^R N_{ij}$$ for $i=1,\dots,R$ and $j=1,\dots,C$. Also, define $n=\sum_{i=1}^R \sum_{j=1}^C N_{ij}=100$.

Let $p_{ij}$ be the cell probabilities. Your null hypothesis is $$H_0 : p_{11}=p_{21}=p_{31}=p_{41} \, .$$

Defining $\hat{E}_{ij}=N_{i\bullet}N_{\bullet j}/n$, it's known that under $H_0$ the statistic $$Q = \sum_{i=1}^R \sum_{j=1}^C {(N_{ij} - \hat{E}_{ij})^2 \over \hat{E}_{ij}} ,$$ has approximately a $\chi^2$ distribution with $(R-1)(C-1)=3$ degrees of freedom.

Hence, a pure significance test would be to reject $H_0$ when $Q>a$, where $a$ is such that $P\{Q>a \mid H_0\}=\alpha_0$, for some chosen significance level $\alpha_0$.

This is called a $\chi^2$ test of homogeneity.

For example, if $\alpha_0=0.05$, then in your case $a$ is approximately $7.81$.

> qchisq(.95, df = 3)
[1] 7.814728


### Edit

For these data, the chi-squared statistic is $2.294$, much less than the $\alpha=0.05$ critical value. More precisely, its "p-value" is $0.5137$: that is, $2.294$ is the $51.37^\text{th}$ percentile of the chi-squared distribution with three degrees of freedom. That tells us these data vary just about as much (or as little) as would be expected under the null hypothesis.

These computations can be carried out with a spreadsheet or hand calculator (along with some chi-squared tables), but there are easier ways. Assuming x is the data matrix above, R gives

> chisq.test(x)

Pearson's Chi-squared test

data:  x
X-squared = 2.2941, df = 3, p-value = 0.5137


Sometimes another test, Fisher's Exact Test, is recommended for analyzing small contingency tables. It is not necessarily better: it tests a similar but subtly different null hypothesis (see the reference for details). Nevertheless, by comparing its p-value to the chi-squared p-value, we can roughly check whether the chi-square approximation is accurate: a large difference would be a red flag. Although there is no reason in these data to suspect that the approximation is not accurate (all the cells have values of $5$ and greater), let's confirm:

> fisher.test(x)

Fisher's Exact Test for Count Data

data:  x
p-value = 0.5126
alternative hypothesis: two.sided


The numbers $0.5126$ and $0.5137$ are so close that we can be comfortable both tests are giving trustworthy p-values: there is no evidence of unreliability in these data.

-
+1 Very good @Zen. This explains in detail the idea that I was getting at in my very brief answer. I agree that the chi-square test for homogeneity and alternatives are exactly what the OP needs to solve his problem. Your answer is a nice little tutorial on this that should be helpful to the OP. The table in your example is exactly how I was suggesting that the OP structure his data and the chi square test you set up and found the critical value for Will work very well to answer the OPs question. He really justs needs to compare his test statistic to this critical value to get his answer. –  Michael Chernick Sep 21 '12 at 9:06
Adding in the p-value would also be helpful I think. I thought of using an exact test first because the marginal totals are small and I was a little afraid that the chi square approximation would not be good. But seeing the table displayed in your answer I can see that every cell is >=5 and the same is probably true for the expected frequency under the null hypothesis for each cell. –  Michael Chernick Sep 21 '12 at 9:12
Zen, do you still have problems viewing the "\frac" in the definition of $Q$? (It looks fine to me.) –  whuber Sep 21 '12 at 14:18
@Zen Given your edit that nicely adds an explanation of Fisher's exact test I think your answer definitely deserves the check mark. Maybe we could add that Fisher's test conditions on the marginal totals and looks at the hypergeometric distribution which under the null hypothesis is associated with the permutations of the table that have the same marginal totals –  Michael Chernick Sep 21 '12 at 16:10
Thank you so much for your detailed response! –  Steve Stewart Sep 22 '12 at 15:26