p-values from t.test and prop.test differ considerably Testing a difference in hits (ones versus zeros) between two independent groups $X$ and $Y$ should be possible with a t-test, according to the following considerations:

*

*$x_i\in\{0,1\}$ is the measurement for the $i$-th item im group $X$, and $y_i\in\{0,1\}$ the same for group $Y$

*the proportion in each group is the mean value of the measurements, i.e. $\mu_X=\sum_i x_i/n$ and $\mu_Y=\sum_i y_i/n$

*the difference of mean values $\mu_X$ and $\mu_Y$ in two groups can be tested with a t-test

In this particular case, a proportion test (function prop.test in R) is an alternative test option. Interestingly, the results are quite different:
> x <- c(rep(1, 10), rep(0, 90))
> y <- c(rep(1, 20), rep(0, 80))
> t.test(x,y,paired=FALSE)
t = -1.99, df = 183.61, p-value = 0.04808
> prop.test(c(10,20), c(100,100))
X-squared = 3.1765, df = 1, p-value = 0.07471

Note the higher p-value of the prop.test. Does this mean that a t-test has a higher power, i.e., can distinguish between $H_0$ and its alternative already for smaller $n$? Is there a reason why a t-test should not be used in this case?
Addition (Edit: resolved in a comment under the answer by Thomas Lumley below): The result of the t-test is even more surprising in the light of the observation that even the asymptotic ("Wald") 95% confidence intervals of both measurements overlap (0.1587989 > 0.1216014):
> library(binom)
> binom.confint(10, 100, method="asymptotic")
      method  x   n mean      lower     upper
1 asymptotic 10 100  0.1 0.04120108 0.1587989
> binom.confint(20, 100, method="asymptotic")
      method  x   n mean     lower     upper
1 asymptotic 20 100  0.2 0.1216014 0.2783986

As confidence intervals based on the t-distribution should be even wider than those based on the normal distribution (i.e. $z_{1-\alpha/2}$), I do not understand why the t-test reports a significant difference at the 5% level.
 A: The t-test can be quite robust to deviations from the normality assumption, particularly when sample sizes are large, so I understand why one might want to use a t-test for this task.
However, you know the parametric family; since the outcome is either $0$ or $1$, the distribution is completely characterized by the relative proportion, thus Bernoulli. Consequently, you can rely on a parametric test designed for a Bernoulli variable, which the t-test is not.
Methods that are robust to deviations from parametric assumptions are wonderful, since we typically do not know the type of population distribution. (If we did, why did we not determine the population parameters when we had the chance!?) However, the case of a binary variable is unique in how it is completely defined by the relative proportion and must be Bernoulli (or easy to represent as Bernoulli, such as calling “heads” and “tails” of a coin $0$ and $1$, respectively).
A: You're correct that the tests should be more similar.  They are tests of means, and for a light-tailed distribution, so you should expect them to agree.  What's more, the estimated variance $\hat p(1-\hat p)/n$ for a binomial distribution is extremely close to $s^2/n$
> var(x)/100
[1] 0.0009090909
> .1*(.9)/100
[1] 9e-04
> .2*(.8)/100
[1] 0.0016
> var(y)/100
[1] 0.001616162

What you're seeing is the continuity correction. If you try it without, the $p$-values are almost identical
> t.test(x,y)

    Welch Two Sample t-test

data:  x and y
t = -1.99, df = 183.61, p-value = 0.04808
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.1991454034 -0.0008545966
sample estimates:
mean of x mean of y 
      0.1       0.2 

> prop.test(c(10,20),c(100, 100),correct=FALSE)

    2-sample test for equality of proportions without continuity correction

data:  c(10, 20) out of c(100, 100)
X-squared = 3.9216, df = 1, p-value = 0.04767
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.197998199 -0.002001801
sample estimates:
prop 1 prop 2 
   0.1    0.2 

The continuity correction for the chi-squared test is a bit controversial. It does dramatically reduce the number of situations where the test is anti-conservative, but at the price of making the test noticeably conservative. Not using the 'correction' gives p-values that are closer to a uniform distribution under the null hypothesis.  And, as you see here, not using the correction gives you something closer to the t-test.
A: *

*A difference between p-values of 0.048 and 0.074 is not large. This can easily happen between tests that don't do exactly the same but a similar thing.


*The theory of the t-test is for normally distributed data, which your data obviously are not. You're right that the t-test can be justified as an approximation, but there's no reason to use an approximation if a more precise test (namely the proportion test) is available. For sure there is no reason to expect the t-test to have a better power, or only in case that it is anticonservative, which is not a good thing (being an approximation, one would probably need to simulate what its finite sample characteristics are in this situation).


*Edited after looking up the reference Agostino et al. ("The Appropriateness of Some Common Procedures for Testing the Equality of Two Independent Binomial Populations", Am. Stat. 1988) given by cdalitz. This reference states that prop.test with continuity correction is too conservative whereas the t-test as well as the prop.test without continuity correction are normally closer to the nominal level, if occasionally anticonservative (which in may view not necessarily justifies an overall recommendation). This was also mentioned in the answer by Thomas Lumley.
If we're ignoring the continuity correction for a moment, there are two differences between the t-test and prop.test (which is not fully documented but I think it does the z-test based on normal approximation).
(a) prop.test uses the knowledge that the variance of the Binomial is $np(1-p)$ rather than using a sample variance based on normality. In my view what prop.test does here should clearly do better, as it is based on information about the specific setup used here.
(b) prop.test uses a normal approximation whereas the t-test uses a t-approximation. Now both of these, applied to the Binomial situation, are asymptotic in nature (the t-distribution is only precise if the underlying data are normal which they aren't here), and actually they are asymptotically equivalent. Although the normal approximation looks more intuitive based on the Central Limit Theorem, this doesn't imply by any means than the normal works better than the t in the finite sample situation (and the t is as well justified by the CLT, if only indirectly). The t-distribution is motivated by the normal assumption, but in fact it may also be the case that the asymptotic normal distribution of prop.test underestimates the finite sample variability because it ignores the variability in the variance estimation, and the t-distribution, despite here not precisely justified, may do a better job at that.
So I now believe that potentially (as could be confirmed by simulations, maybe somebody has done that?) the best thing to do could be using the test statistic of prop.test, i.e., the "correct" variance estimation, but replacing the asymptotic normal distribution by a t-distribution, which in some sense may put together the advantages of them both.
