Choosing between $z$-test and $t$-test

Question

Background: I'm giving a presentation to colleagues at work on hypothesis testing, and understand most of it fine but there's one aspect that I'm tying myself up in knots trying to understand as well as explain it to others.

This is what I think I know (please correct if wrong!)

• Statistics that would be normal if variance was known, follow a $t$-distribution if the variance is unknown
• CLT (Central Limit Theorem): The sampling distribution of the sample mean is approximately normal for sufficiently large $n$ (could be $30$, could be up to $300$ for highly skewed distributions)
• The $t$-distribution can be considered Normal for degrees of freedom $> 30$

You use the $z$-test if:

1. Population normal and variance known (for any sample size)
2. Population normal, variance unknown and $n>30$ (due to CLT)
3. Population binomial, $np>10$, $nq>10$

You use the $t$-test if:

1. Population normal, variance unknown and $n<30$
2. No knowledge about population or variance and $n<30$, but sample data looks normal / passes tests etc so population can be assumed normal

So I'm left with:

• For samples $>30$ and $<\approx 300$(?), no knowledge about population and variance known / unknown.

So my questions are:

1. At what sample size can you assume (where no knowledge about population distribution or variance) that the sampling distribution of the mean is normal (i.e. CLT has kicked in) when the sampling distribution looks non-normal? I know that some distributions need $n>300$, but some resources seem to say use the $z$-test whenever $n>30$...

2. For the cases I'm unsure about, I presume I look at the data for normality. Now, if the sample data does looks normal do I use the $z$-test (since assume population normal, and since $n>30$)?

3. What about where the sample data for cases I'm uncertain about don't look normal? Are there any circumstances where you'd still use a $t$-test or $z$-test or do you always look to transform / use non-parametric tests? I know that, due to CLT, at some value of $n$ the sampling distribution of the mean will approximate to normal but the sample data won't tell me what that value of $n$ is; the sample data could be non-normal whilst the sample mean follows a normal / $t$. Are there cases where you'd be transforming / using a non-parametric test when in fact the sampling distribution of the mean was normal / $t$ but you couldn't tell?

Answer 1

@AdamO is right, you simply always use the $t$-test if you don't know the population standard deviation a-priori. You don't have to worry about when to switch to the $z$-test, because the $t$-distribution 'switches' for you. More specifically, the $t$-distribution converges to the normal, thus it is the correct distribution to use at every $N$.

There is also a confusion here about the meaning of the traditional line at $N=30$. There are two kinds of convergence that people talk about:

1. The first is that the sampling distribution of the test statistic (i.e., $t$) computed from normally distributed (within group) raw data converges to a normal distribution as $N\rightarrow\infty$ despite the fact that the SD is estimated from the data. (The $t$-distribution takes care of this for you, as noted above.)
2. The second is that the sampling distribution of the mean of non-normally distributed (within group) raw data converges to a normal distribution (more slowly than above) as $N\rightarrow\infty$. People count on the Central Limit Theorem to take care of this for them. However, there is no guarantee that it will converge within any reasonable sample size--there is certainly no reason to believe $30$ (or $300$) is the magic number. Depending on the magnitude and nature of the non-normality, it can take very long (cf. @Macro's answer here: Regression when the OLS residuals are not normally distributed). If you believe your (within group) raw data are not very normal, it may be better to use a different type of test, such as the Mann-Whitney $U$-test. Note that with non-normal data, the Mann-Whitney $U$-test is likely to be more powerful than the $t$-test, and can be so even if the CLT has kicked in. (It is also worth pointing out that testing for normality is likely to lead you astray, see: Is normality testing 'essentially useless'?)

At any rate, to answer your questions more explicitly, if you believe your (within group) raw data are not normally distributed, use the Mann-Whitney $U$-test; if you believe you data are normally distributed, but you don't know the SD a-priori, use the $t$-test; and if you believe your data are normally distributed and you know the SD a-priori, use the $z$-test.

It may help you to read @GregSnow's recent answer here: Interpretation of p-value in comparing proportions between two small groups in R regarding these issues as well.

Answer 2

There's nothing to discuss on the matter. Use a $t$-test always for a nonparametric test of differences in means, unless a more sophisticated resampling tool—e.g. permutation or bootstrap—is called for (useful in very small samples with large departures from normality).

If the degrees of freedom actually matter, then the $t$-test will provide consistent estimation of critical values and standard errors for the distribution of the test statistic under the null hypothesis. Otherwise, the $t$-test is approximately the same as the $z$-test.

The normal approximation to tests of parametric model parameters, like the population proportion test, is kind of defunct. When the data are small enough that there really is a distinction between critical values generated from $t$ or $z$ distributions, you really should be using an exact test of proportions based on the scaled binomial distribution of the test statistic. Resampling tests work this way as well. Making arbitrary rule-of-thumb assumptions about sample sizes and prevalence of cases/controls in estimation of Bernoulli parameters is confusing and extremely error prone.

The concept of a $z$-test ("known" variance) is confusing because you never "know" variance, nor do you spend much to estimate it. When that cost matters, only the $t$-test reflects its impact upon the degrees of freedom.