15
$\begingroup$

(Disclaimer: I am aware that some form of this question has been asked many times on this site, e.g. this, this and this - I was hoping to find a definitive answer to this from Google but I am still confused by reading these links and I don't have enough privilege to comment, so here goes this question.)

Mathematically, it is necessary to assume data is normal, for t-test to be applied. (This is because of the denominator in t-statistic, where we assume sample variance distributes like chi squared after normalization)

  • The most popular answer to justify "t-test is good enough for non-normal population" is to say that for large sample sizes, the sample mean is approximately normal by Central Limit Theorem. There is no guarantee on how large the sample size is needed, but we can take it on faith.
  • I'd argue that by the exact same argument, we can just use z-test instead of t-test, especially since t-distribution is close to normal when n is "big", so Slutsky's Theorem would apply.
  • This must mean that if we favor the t-test over the z-test, we must believe that in a transitional range where n is not big enough, the t-test is better than the z-test in some way, even for non-normal data (e.g. the t-test is more powerful than the z-test). Why? What are the assumptions that go into such statement, or is it purely empirical?
$\endgroup$
4
  • 5
    $\begingroup$ Welcome to CV, dummy. I like the way your question is asking after a rigorous defense of a common heuristic. $\endgroup$
    – Alexis
    Commented Sep 25, 2022 at 18:55
  • 6
    $\begingroup$ Normality assumptions aside, the choice between the $t$-test and $z$-test depends on whether the standard deviation is known ($z$) or has to be estimated from the data ($t$). When the standard deviation is estimated, the test statistic has a $t$-distribution (ceteri paribus) at any sample size. If it is know, the test statistics is normally distributed, also at any sample size. It is simply that at larger samples, the difference becomes negligible. $\endgroup$ Commented Sep 25, 2022 at 19:02
  • $\begingroup$ Hi @COOLSerdash, why is "the test statistic has a 𝑡-distribution (ceteri paribus) at any sample size" true for non-normal data? $\endgroup$
    – dummy
    Commented Sep 25, 2022 at 20:24
  • $\begingroup$ It's not true for non-normal data. As you mention, you only get the $t$-distribution under normality. $\endgroup$ Commented Sep 25, 2022 at 21:17

2 Answers 2

23
$\begingroup$

Technically your argument is correct. The t- and z-test are asymptotically equivalent. However, the t-test takes into account the variation in the estimation of the standard error. Obviously, if data are non-normal, it cannot be shown that the distribution of the test statistic is a t-distribution. However, the t-distribution has heavier tails than the normal, so the t-test will be more conservative than the z-test, and the z-test may for finite samples well be anti-conservative (due to treating the standard deviation as fixed and known , which it isn't), so the t-test will be safer regarding type I error probabilities.

I believe that when simulating from specific distributions one may find occasionally that the z-test does better (i.e., better power while respecting the level), however I believe that the t-test will be better (respecting the level while the z-test is anti-conservative, or being closer to the level if both are anti-conservative) far more often. If I remember correctly I have once simulated one particular case with the t-test winning, however I didn't do anything comprehensive. I'm pretty sure you won't find a general theorem as the t-test is not universally better.

To add some intuition to this, I believe that it is generally correct that estimation of the standard error should lead to heavier tails of the distribution of the test statistic, so that in most cases the normal distribution will underestimate variation and the heavier tails of the t-distribution are more appropriate. However there may be distributions for which the finite sample distribution of the mean is subgaussian, i.e., has lighter tails than the normal (even though converging to the normal by means of the CLT). Estimation of the standard deviation in such cases will still make the tails heavier than treating it as fixed, but the normal approximation may still keep the test level in case it was overconservative enough with fixed sd. In such a (probably exceptional) case the z-test will yield better power, so it may be better. With the t-test however we are always on the safer side regarding type I error.

Note that all this assumes that (at least) second moments exist. If they don't (or if in practice tails are very heavy or there are extreme outliers), variance estimation in particular is a mess (also mean estimation but not quite as much), and both tests won't work well.

$\endgroup$
5
  • $\begingroup$ Thanks for your answer! Some follow up questions to make sure I follow: 1. When you say these assume second moments exist - is your point that CLT needs to be valid for the t-statistic to approximately follow z-distribution or t-distribution? It seems true that no matter what, t-test is more conservative than z-test. 2. While I can buy the "t test is safer for type 1 error" argument, it seems unnecessary to do such adjustment using t-distribution. If I believe approximation by normal distribution increased type 1 error, I can also just increase my significance level. (It's not clear what ... $\endgroup$
    – dummy
    Commented Sep 25, 2022 at 20:44
  • $\begingroup$ the "new" significance level should be and may be arbitrary, but using t-distribution for such adjustment feels arbitrary as well, other than t-test being a "well-established test") Any thoughts about this? $\endgroup$
    – dummy
    Commented Sep 25, 2022 at 20:46
  • $\begingroup$ @dummy (1) The CLT needs second moments. If you don't have that, in fact the power may suffer more than the effective level (which may get too conservative), so more conservativity may not necessarily be better. The z-test may have rubbish power and the t-test even worse, although this will depend on the specific distribution. (2) Adjustment of the significance level will not normally help as (anti-)conservativity is defined relative to the nominal significance level, and it may strike just as much with a different level. $\endgroup$ Commented Sep 25, 2022 at 21:02
  • $\begingroup$ @dummy Of course you might say, I want to test at 5%, so because of anti-conservativity I test at nominal 4%, but such an adjustment will depend on what the true underlying distribution is, which you don't know, so nobody can tell you how to adjust. So the standard is to interpret at the nominal level, and then of course the t-test is better to the extent that it respects the nominal level better. $\endgroup$ Commented Sep 25, 2022 at 21:06
  • $\begingroup$ Thanks! Agree with 1. Sort of agree with 2 (though I see what you're saying; especially your added remark) - what I meant was: assuming "95%" is the significance level I'm taking, whatever that means. If I believe using a normal approximation undercounts type 1 error, I may as well use, say, a 98% significance level for z-test to still represent the same "95% sig level" in my mind. To me, using t-distribution instead of z-distribution as an approximation does the same thing, because 95% sig level for t-test means something slightly different for 95% sig level for z-test. $\endgroup$
    – dummy
    Commented Sep 25, 2022 at 21:08
0
$\begingroup$

Now this isn't going to be a very rigorous answer, but take it for what it is.

I want to distinguish between two things. There is a difference between your data, and the statistics that you compute from your data.

T-tests are for statistics, not the data itself. The statistics will be normally distributed. Plenty of proofs of the CLT that we could refer to. It is this fact that you are leaning on that makes this work. But wait, why are we using t-tests if I said that your statistic is guaranteed to be normally distributed.

The answer is that you are using your statistic to compute a new statistic about itself. This new secondary statistic is the variance of your guaranteed normal distribution. It is an estimate not of the variance of the data, but of the statistic. This is known as a standard error.

Now, you have two sources of error about the distribution for your statistic.

  1. The original sampling error from the population when you compute your statistic
  2. The additional error from estimating the variance of your statistic utilizing your statistic.

You use a t-test to adjust for the uncertainty coming from source 2 of your error. The bigger your samples are, the less variance you expect will come from source 2, and your sample will approach a z-test, which would only consider errors coming from source 1, which you can't get rid of.

Essentially, doing a t-test is an act of humility where you are admitting that errors may be creeping into your confidence intervals because of the way that you are computing them. There is no way to get rid of these potential errors, so we are going to make them slightly larger than you would see with a z-test. Of course, there are derivations for why the t-distribution is the correct distribution to use to do this and so forth (I won't subject myself and you to the same tortures I did in school with gamma functions but you can look them up if you are brave enough), but that is the intuitive "why".

$\endgroup$
4
  • $\begingroup$ Thanks for your answer. I am not sure if I buy "you need to account for the variance being estimated in some way", because 1. Using t-distribution to do this accounting assumes that your data (not sample mean) follows a normal distribution. In other words, it's not clear why this kind of "accounting of variance being estimated" works for non-normal data, and 2. if we can hand wave sample mean being normal by CLT, why can't we hand wave estimated variance being the same as actual variance when n is big? $\endgroup$
    – dummy
    Commented Sep 27, 2022 at 0:42
  • $\begingroup$ I think that you misunderstand. The accounting happens because you use the mean to estimate the variance. Point 1) you are testing whether a mean is different from some number or some other mean with a t-test. You are not comparing the data and its distribution to something else. If you were to do that something like a ks-test would be more appropriate. 2) You actually can, "handwave" because the estimated variance will also follow a normal distribution by the CLT. The trick though is that as your sample size approaches the population size, your uncertainty about the true variance goes to zero $\endgroup$
    – Ryan
    Commented Sep 28, 2022 at 18:50
  • $\begingroup$ The big issue is that the data themselves need not be normally distributed because what you are implicitly assuming is that you are going to sample from the data multiple times with replacement, calculate the mean of each of your samples and generate a new dataset, a dataset of means. This is what your t-distribution is looking at. You are comparing the mean you calculated to the distribution you get from this sampling procedure. Again, you are comparing the mean to essentially another mean. Therefore, you want to look at the distribution of means, not the distribution of the data. $\endgroup$
    – Ryan
    Commented Sep 28, 2022 at 19:00
  • $\begingroup$ I think this is the fundamental idea of inferential statistics (in general), and it isn't particularly well communicated, or understood for that matter. It's why people don't understand why bootstrapping procedures work. $\endgroup$
    – Ryan
    Commented Sep 28, 2022 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.