I am trying to figure out what test I should use in the following scenario: I know that there is a lot of room for improvement in a specific area at work - being extremely critical, let's say that sampling $52$ observations, $31$ could be improved. After instituting an improvement / QA program for six months, let me assume that out of a sample of $55$ cases, there are only $11$ with residual flaws. The two samples are independent. We are therefore comparing two proportions: $p_{\text{ initial}} =\frac{31}{52}$ and $p_{\text{ final}} = \frac{11}{55}$.
Although the numbers are exaggerated, I still want to see if the two proportions are statistically significantly different, and I think I have a couple of options: I can run an exact binomial test to calculate the probability that the new proportion of flawed observations, $\frac{11}{55}$, would occur if the actual underlying probability remained $\frac{31}{52}$. Alternatively, I can run a chi-squared test.
The chi-squared is an approximation, and what I have read is that it is to be applied when the total number of observations is too high. This is clearly not the case in the example; however, playing with the numbers in R, I couldn't see any delay or problems with the results even after using numbers $>10,000$. And there was no indication of any normal approximation being used.
So, if this is all true, why shouldn't we always opt for an exact binomial test, rather than a chi square?
The code in R for the two test would be:
# Exact Binomial Test:
binom.test(c(11, 55 - 11), p = 31/52, alternative ="less")
#Chi-square Test:
prop.test(c(31, 11), c(52, 55), correct = FALSE, alternative = 'greater')
binom.test
seems inappropriate here. You need to compare two datasets, not one dataset to a fixed probability. Setting $p=31/52$ ignores the uncertainty in the estimated value of $31/52$ for the pre-intervention rate and thereby (substantially) increases the false positive error rate. $\endgroup$