Testing a difference in hits (ones versus zeros) between two independent groups $X$ and $Y$ should be possible with a t-test, according to the following considerations:
- $x_i\in\{0,1\}$ is the measurement for the $i$-th item im group $X$, and $y_i\in\{0,1\}$ the same for group $Y$
- the proportion in each group is the mean value of the measurements, i.e. $\mu_X=\sum_i x_i/n$ and $\mu_Y=\sum_i y_i/n$
- the difference of mean values $\mu_X$ and $\mu_Y$ in two groups can be tested with a t-test
In this particular case, a proportion test (function prop.test
in R) is an alternative test option. Interestingly, the results are quite different:
> x <- c(rep(1, 10), rep(0, 90))
> y <- c(rep(1, 20), rep(0, 80))
> t.test(x,y,paired=FALSE)
t = -1.99, df = 183.61, p-value = 0.04808
> prop.test(c(10,20), c(100,100))
X-squared = 3.1765, df = 1, p-value = 0.07471
Note the higher p-value of the prop.test. Does this mean that a t-test has a higher power, i.e., can distinguish between $H_0$ and its alternative already for smaller $n$? Is there a reason why a t-test should not be used in this case?
Addition (Edit: resolved in a comment under the answer by Thomas Lumley below): The result of the t-test is even more surprising in the light of the observation that even the asymptotic ("Wald") 95% confidence intervals of both measurements overlap (0.1587989 > 0.1216014):
> library(binom)
> binom.confint(10, 100, method="asymptotic")
method x n mean lower upper
1 asymptotic 10 100 0.1 0.04120108 0.1587989
> binom.confint(20, 100, method="asymptotic")
method x n mean lower upper
1 asymptotic 20 100 0.2 0.1216014 0.2783986
As confidence intervals based on the t-distribution should be even wider than those based on the normal distribution (i.e. $z_{1-\alpha/2}$), I do not understand why the t-test reports a significant difference at the 5% level.
correct=FALSE
in the chi-squared you will see the p-value is fairly close to that of the t-test, or if you compute the continuity-corrected t-test and compare with the above chi-squared, you'll see the two fairly close again. $\endgroup$correct=FALSE
) has an actual $\alpha$ probability much closer to the nominal level than the corrected chi square test. The t test is thus perfectly ok in this case, and my question is answered. $\endgroup$