14
$\begingroup$

Given some standard assumptions, the test statistic

$$ \frac{\Delta\bar{X}}{\sigma/\sqrt{N}} $$

is normally distributed if $\sigma$ is known and t-distributed if $\sigma$ has to be estimated from the sample. The difference of the distributions is most pronounced for small sample sizes and in the tails (which are fatter for the t-distribution).

I wonder whether there's an intuitive explanation as to why the tails of the t-distribution are fatter when $\sigma$ is not (exactly) known.

I found two intuitive explanations for the fat tails of the t-distrbution on CV.

The first one:

The overall answer is that the T-Distribution gives a higher probability to extreme events, given a small sample size. The reasoning behind this is intuitive in that, as you can imagine, if you have relatively small sample from a population, there is a higher probability that an extreme event from the population did not "make it" into your sample. On the other hand, if you have a relatively large sample from the same population, the probability that an extreme event did not "make it" into your sample is lower, since your sample is capturing more of that population.

This is the second one:

With small sample sizes, it's relatively likely that one of the two samples will receive a slightly extreme datapoint that will skew its mean and make you think that there is a real difference between the two samples when instead nothing is going on. To correct for such small-size effect you allow more extreme values of the difference between samples to be relatively likely before claiming "significance of difference". I.e. the t-distribution, against which you map the value of the t-statistics to obtain the p-value, has fatter tails than the normal distribution. As the sample sizes increases, this correction becomes negligible and you don't need to worry about using Normal or T-distribution (or T-test vs Z-test).

Now, it seems to me that the two explanations are contradictory (though not 100% sure) and to me the second explanation is very intuitive.

In any case, both explanations would be compatible with a scenario of known population variance as well. They do not even talk about known/unknown population variance, even though if it is known we would (theoretically) obtain a normal distribution even for small samples.

So my question: is there an intuitive explanation as to why the tails of the t are (need to be) fatter precisely when $\sigma$ is unknown? In particular, why does the argument given in the quoted second explanation only apply to a case of unknown variance?

$\endgroup$
4
  • 4
    $\begingroup$ +1 I find both attempted explanations to be lacking (to be polite about it), because on the face of it they apply directly to the standard Z test, which obviously leads to a self-contradictory "explanation." The entire point is that the estimated standard deviation has a chance of being appreciably less than the population SD and despite indicating that the data are unlikely to be extreme, that inflates the chances of extreme t statistics. That's still no explanation, because it's not at all obvious what this division ought to do to the tails. $\endgroup$
    – whuber
    Commented Sep 5, 2023 at 14:34
  • 3
    $\begingroup$ See stats.stackexchange.com/questions/110359/… ... there's an answer that explains the fatter tails in some detail, as well as why it changes with sample size $\endgroup$
    – Glen_b
    Commented Sep 5, 2023 at 17:25
  • $\begingroup$ Thanks @Glen_b, this nicely generalizes the conclusion of the simulation in the accepted answer! $\endgroup$
    – monade
    Commented Sep 5, 2023 at 19:22
  • $\begingroup$ The first explanation have some value, the second is completely off-point. $\endgroup$ Commented Sep 6, 2023 at 13:07

1 Answer 1

17
$\begingroup$

The heavy tails arise because in small samples, standard errors near zero have appreciable chances of arising and they are not associated with concomitantly small sample means that would cancel them out in the Student t statistic.

To appreciate this, consider small samples from a uniform distribution (which produces no extreme values of any kind). Here, from a simulation of samples of size two, are plots of the absolute value of the mean $|m|$ and standard error $se:$ enter image description here

The points in the scatterplot are colored according to the inverse slopes, $t = m/se,$ which are the Student t statistics. The contour lines of constant slope are, of course, lines through the origin. Near the base the inverse slopes are nearly infinite. Notice the pattern:

Large standard errors are associated with "typical" sample means (near $1/2$), where (accordingly) $t$ is not large. Conversely, the only way for $t$ to grow large is along the base of the triangle away from the origin where (a) the standard error is tiny but (b) the mean is not tiny.

This picture is not peculiar to the uniform distribution: it's just plainer than the Normal case, shown next.

enter image description here

(The horizontal line in the scatterplot is the expected standard error $\sigma/\sqrt{n}$ you would use for a Z statistic.)

As is well known, the standard error is uncorrelated with the mean. Nevertheless, it is still the case that

The only way for $t$ to grow large (with any appreciable probability) is along the base of the triangle away from the origin at its center where (a) the standard error is tiny but (b) the mean is not tiny.

That is, the heavy tails of the Student t distribution arise from the chance that a $(m, se)$ pair will be situated near the bottom left or bottom right of the scatterplot. In those cases the division by a near-zero standard error yields a huge value of the Student t statistic.


In retrospect, this is obvious in samples of size 2, because there's a $1/2\times 1/2=1/4$ chance that both values in a sample will have the same sign. Because they have the same sign, likely they are closer together than your average sample: there are your principal contributions to the heavy tails. The effect holds in large samples but quickly fades away once it's unlikely for the entire sample to have a small range.

$\endgroup$
8
  • $\begingroup$ in case you have the understanding r script, would you disclose as part of answer? It is always good to have the full replicability available to learn from. Many thanks. $\endgroup$
    – Maximilian
    Commented Sep 5, 2023 at 17:39
  • 1
    $\begingroup$ @Max Apart from the plotting functions, which are routine applications of density and plot, the simulation code to create the sample means m and standard errors s is simply n <- 2; X <- matrix(runif(n*1e5), n); m <- colMeans(X); s <- apply(X, 2, sd) / sqrt(n) Replace runif by rnorm for the second example. $\endgroup$
    – whuber
    Commented Sep 5, 2023 at 17:49
  • 2
    $\begingroup$ Thanks a lot, this is super helpful. I particularly like the example of the uniform distribution, as it directly counters the point about the effect of extreme data points on the mean. So, in combination with the information in the link provided by @Glen_b in the comments, is it fair to say that the fat tails of the t-distribution are a direct consequence of the right skew of the chi distribution of $\hat{\sigma}$ at low sample sizes? $\endgroup$
    – monade
    Commented Sep 5, 2023 at 19:20
  • 1
    $\begingroup$ I see it a little differently. The heavy tails arise due to the chance that a chi variable (proportional to the SE for a Normal mean) is close to zero. The right way to examine it is on a log scale (for its argument), not the usual scale -- and on the log scale, the distribution is left skewed. In other words, there is an imbalance towards dividing by small numbers compared to dividing by large numbers. Once the chi degrees of freedom grow large -- say, around 30 or so -- the distribution appears symmetric on the log scale. $\endgroup$
    – whuber
    Commented Sep 5, 2023 at 20:05
  • $\begingroup$ I'm not sure I understand. "The heavy tails arise due to the chance that a chi variable is close to zero" was exactly the argument when I said that the fat tails are a consequence of the right skew of the chi distribution. I'm also confused by your statement about the log-scale: if it is left-skewed shouldn't there be an imbalance away from small values (on the log scale)? $\endgroup$
    – monade
    Commented Sep 5, 2023 at 21:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.