How do I reject or fail to reject the null hypothesis that p1 = p2 = p3 = p4? I need to do reliability analysis on computerized test questions.  Given a multiple choice question, if 20/25 got it right the first time the test was given, 22/30 the second time the test was given, 15/22 the third time the test was given, and 14/23 the fourth time the test was given, I want to determine if all of these came from the same population or not.  [I assume the population is the same, but is the question reliable?]  If I fail to reject the null hypothesis the assumption is reliability, although technically not proven. 
I am thinking of using ANOVA on this for proportions, but I don't know that the necessary assumptions have been met to do this.  My sample sizes are small. What do you recommend?  
 A: You have a contingency table with $C=2$ columns and $R=4$ rows.
       Right    Wrong
1st     20        5       25
2nd     22        8       30
3rd     15        7       22
4th     14        9       23
        71       29      100

Let $N_{ij}$ be the number of observations in the $(i,j)$-cell. Define
$$
    N_{i+}=\sum_{j=1}^C N_{ij} \qquad \textrm{and} \qquad N_{+j}=\sum_{i=1}^R N_{ij}
$$
for $i=1,\dots,R$ and $j=1,\dots,C$. Also, define $N_{++}=\sum_{i=1}^R \sum_{j=1}^C N_{ij}=100$.
Let $p_{ij}$ be the cell probabilities. Your null hypothesis is
$$
  H_0 : p_{11}=p_{21}=p_{31}=p_{41} \, .
$$
Defining $\hat{E}_{ij}=N_{i+}N_{+j}/N_{++}$, it's known that under $H_0$ the statistic
$$
  Q = \sum_{i=1}^R \sum_{j=1}^C {(N_{ij} - \hat{E}_{ij})^2 \over \hat{E}_{ij}} ,
$$
has approximately a $\chi^2$ distribution with $(R-1)(C-1)=3$ degrees of freedom. 
Hence, a pure significance test would be to reject $H_0$ when $Q>a$, where $a$ is such that $P\{Q>a \mid H_0\}=\alpha_0$, for some chosen significance level $\alpha_0$.
This is called a $\chi^2$ test of homogeneity.
For example, if $\alpha_0=0.05$, then in your case $a$ is approximately $7.81$.
> qchisq(.95, df = 3)
[1] 7.814728

Edit
For these data, the chi-squared statistic is $2.294$, much less than the $\alpha=0.05$ critical value.  More precisely, its "p-value" is $0.5137$: that is, $2.294$ is the $51.37^\text{th}$ percentile of the chi-squared distribution with three degrees of freedom.  That tells us these data vary just about as much (or as little) as would be expected under the null hypothesis.
These computations can be carried out with a spreadsheet or hand calculator (along with some chi-squared tables), but there are easier ways.  Assuming x is the data matrix above, R gives
> chisq.test(x)

        Pearson's Chi-squared test

data:  x 
X-squared = 2.2941, df = 3, p-value = 0.5137

Sometimes another test, Fisher's Exact Test, is recommended for analyzing small contingency tables.  It is not necessarily better: it tests a similar but subtly different null hypothesis (see the reference for details). Nevertheless, by comparing its p-value to the chi-squared p-value, we can roughly check whether the chi-square approximation is accurate: a large difference would be a red flag.  Although there is no reason in these data to suspect that the approximation is not accurate (all the cells have values of $5$ and greater), let's confirm:
> fisher.test(x)

    Fisher's Exact Test for Count Data

data:  x 
p-value = 0.5126
alternative hypothesis: two.sided 

The numbers $0.5126$ and $0.5137$ are so close that we can be comfortable both tests are giving trustworthy p-values: there is no evidence of unreliability in these data.
A: ANOVA is not appropriate in this case.  Think of this as a 2x4 contingency table.  I think the generalization of Fisher's exact test to RxC tables will do exactly what you want. Zen has provided detailed calculations for the Fisher test using program in R (I think it is in R).  His proposed chi square approximate test works fine in your application.
