Interpretation of p-value in comparing proportions between two small groups in R

Question

I wish to compare proportions between two small groups.

In group 1, 2 of 18 students received honours. In group 2, 9 of 21 received honours.

This is a small sample, so if I use the t-test:

x1 <- c(rep(1,2),rep(0,16))
x2 <- c(rep(1,9),rep(0,12))
t.test(x1,x2, alternative="two.sided", conf.level = 0.95)

I obtain a p-value of .0239 and CI of (-0.590, -0.0445):

If I had used the z-test (a.k.a. chi-squared test with 1 degree of freedom):

prop.test(x=c(2,9), n= c(18,21), alternative="two.sided", conf.level=0.95)

I get a p-value of .0659 and a CI of (-0.626, -0.00921)

I had always thought that for small samples one must use a t-test since the normal approximation is not valid for small samples, causing exaggerated significance under z-test. So why is the difference significant under the t-test, but not under the z-test? And in the t-test, why does the CI not include zero when the p-value is less than alpha?

Answer 1

You are mixing a couple of concepts (one of which is really outdated in the age of computers, but still persists anyways).

For numerical variables you should use the t-test when you are calculating the standard deviation(s) from the sample(s). If you know the true population standard deviation(s) then you can use the z-test (it is rare that you would know the standard deviation(s) but not the mean(s)). Before computers were easily available for doing statistics, we used tables to look up p-values or critical values. Some t-tables only went up to about 30 degrees of freedom since the values for more than 30 degrees of freedom are almost the same as the normal. This lead to the rule of using a z-test for large sample sizes (larger than your table went to) and the t-test for smaller values (that could be looked up on the t-table). But with modern computers and statistical software we can calculate the t values for any number of degrees of freedom. So we should always use the t-test for numerical variables when we do not know the population standard deviation and only use the z-test for the rare cases where we do know (or can assume from the null hypothesis) the population standard deviation.

When working with proportions the standard deviation is a function of the proportion and is therefore "known", so we never use the t-test for proportions. If the sample size is large enough relative to the proportions being tested then the binomial can be approximated by a normal distribution, so for large sample you can use the z-test. This can lead to the confusion (not unique to you). But for small samples on proportions you need to use a different test that is based on the binomial or hypergeometric distributions. One option is Fisher's exact test:

> fisher.test( cbind( c(2,9), c(18-2, 21-9) ) )

Fisher's Exact Test for Count Data

  data:  cbind(c(2, 9), c(18 - 2, 21 - 9))
  p-value = 0.03749
  alternative hypothesis: true odds ratio is not equal to 1
  95 percent confidence interval:
  0.01557731 1.06612067
  sample estimates:
  odds ratio 
   0.1744356 

The confidence interval here is on the odds ratio rather than the difference of proportions.