The $z$-test vs the $\chi^2$-test for comparing the odds of catching a cold in 2 groups

Question

I just read in a rather well-respected (popular) science magazine (the German PM, 02/2013, p.36) about an interesting experiment (without a source, unfortunately). It caught my attention because intuitively I doubted the significance of the result, but the information provided was sufficient for reproducing the statistical testing.

The researchers wondered whether getting cold in cold weather increases the odds of catching a cold. So they randomly split a group of 180 students into two groups. One group had to hold their feet into cold water for 20 minutes. The other kept their shoes on. Kind of a funny manipulation, I think, but on the other hand I am not a doctor and maybe doctors think funny. Ethical issues aside.

Anyways, after 5 days, 13 of the students in the treatment group had a cold, but only 5 in the group that kept their shoes on. The odds ratio of this experiment thus is 2.87.

Given the rather small sample size, I started wondering if this difference may be significant. So I conducted two tests.

First a simple test of equality of proportions using the normal approximation. This test has $z=1.988$ with $p=0.0468$. My guess is that this is what the researchers tested. This is truely just significant. However this z-test is only valid in large samples, if I am not mistaken, due to the normal approximation. Furthermore, the prevalence rates are rather small and I wonder whether this may not affect the coverage rate of the confidence interval of the effect.

So my second try was a chi-square test of independence, both with Monte-Carlo simulation and standard Pearson Chi-square. Here I find p-values both about $p=.082$.

Now that's all not so reassuring about the results. I wondered if there are more options to test this data and what your thoughts on the two tests are (in particular the assumptions of the first, significant, test)

Answer 1

I'd use a permutation test instead of either the Normal approximation or the chi-square. The permutation test is exact and most powerful, conditional upon the data.

In this case, we can't calculate all the permutations of the groups, but we can generate a lot of random permutations of the data and get a pretty precise value:

group <- c(rep("A",90),rep("B",90))
n_a <- rep(0,100000)
for (i in 1:length(n_a)) {
   temp <- sample(group, size=18)
   n_a[i] <- sum(temp == "A")
}
> mean(n_a >= 13)
[1] 0.03904

which would indicate a p-value of 0.039.

HOWEVER, and this is a big however, I'm guessing that the assumption that the subjects getting colds are independent events is violated. These individuals are students, presumably at the same school. Imagine two of them share a class, or a dorm, or some other activity, or a cafeteria (in a school with multiple cafeterias); the events "#1 gets a cold" and "#2 gets a cold" are not independent. I could imagine that a student would say "let's sign up for this experiment!" to his/her roommate or friends; I could imagine that the students were recruited from classes that the professors taught; I could imagine a lot of ways that the assumption of independence is violated. Perhaps the paper, which I have not read, addresses some of these, but it's hard to see how it could address all of them, and the others that come quickly to mind.

Answer 2

@jbowman has given you a good option. I thought I might provide some information regarding your explicit questions about the appropriateness of the $z$-test vs. the $\chi^2$ test.

$\boldsymbol z$-test:

There are two concerns about the appropriateness of using the $z$-test, both regarding whether the assumed sampling distribution is correct. First, the $z$-test uses the normal distribution instead of the $t$-distribution, implying the standard deviations are known without sampling error. Second, the sampling distribution is continuous, but the data are discrete; since only certain combinations of data are possible, only certain resulting realized test statistic values are possible, which may not well match the theoretical sampling distribution. (I discuss this issue in the context of other tests here: Comparing and contrasting, p-values, significance levels and type I error.)

Let's consider the first concern in a different context. If you have two groups with normally distributed data, and you want to see if the means are equivalent, you need to calculate both the means and the standard deviations. Now we know that the means are subject to sampling error, that's why we need to do the test rather than just say these two sample means aren't identical. However, our estimates of the standard deviations also have to be subject to sampling error and we have to take that fact into account somehow. When we do that, it turns out that the test statistic (a kind of scaled mean difference) is distributed as $t$. If we used the normal distribution instead (i.e., the $z$-test), it would mean that we are assuming that our estimates of the standard deviations are without error--perfect. So why could the $z$-test be used in your case? The reason is that your data are binomial (i.e., the number of 'successes' out of a known total of 'trials'), instead of normal. In the binomial distribution, the standard deviation is a function of the mean, so once you have estimated the mean there is no additional uncertainty to have to worry about. Thus, the normal distribution can be used as a model of the sampling distribution of the test statistic.

Although using the normal distribution to understand the long-run behavior of the test statistic is technically correct, another issue emerges. The problem is that the normal distribution is continuous, but because your data are discrete, not all values in the theoretical distribution can possibly be found in your dataset. (Again, I discuss this issue in considerably more detail in the above linked answer.) Fortunately, the match between the possible outcomes of your data and the theoretical normal sampling distribution gets better the larger your $N$. In your case, no matter what the true underlying probabilities, you could have as many as all successes or as few as none in each group. That means the number of possible combinations is $91\times 91 = 1,\!729$, which is a lot of possibilities. With a small dataset, you really can run into some of the kinds of problems I discuss in my linked answer, but with $N = 180$, you don't have too much to worry about. I believe the $z$-test was a valid choice for the researchers.

$\boldsymbol \chi^2$-test:

But what about the $\chi^2$-test? I think that is also a valid choice, but it wouldn't be my first choice. (Let me note in passing that the second concern discussed above--a mismatch between discrete data and a continuous reference distribution--applies just as much to the $\chi^2$-test as it does to the $z$-test, so there is no advantage here.) The problem with the $\chi^2$-test is that it doesn't assume there is anything special about the column totals relative to the row totals; both are treated as though they could have been other possible values. However, this does not accurately reflect the experimental setup. There were 180 people, and 90 were assigned to each group. The only thing that would truly vary across repeated identical studies is the number of people who caught a cold in each group. The $\chi^2$-test incorrectly treats both the number of colds and the number of people in each group as though they could vary, but the $z$-test makes the right assumption. That's why the $z$-test has more power here.

For what it's worth, the permutation test suggested by @jbowman also gets this aspect of your design right and doesn't suffer from the discrete-continuous mismatch issue. Thus, it is the best option. But I thought you might like to know a little bit more about how the $z$- and $\chi^2$-tests compare in your situation.