1
$\begingroup$

this PSU master's degree course says t-test can be used if either of the conditions is true:

  1. population is normal.
  2. sample size n>30.

However, I read that t test is applicable when $X_{i}\stackrel{iid}{\sim}N(\mu,\sigma^{2})$, because this leads to the independence between $a=\frac{\bar{X}-\mu_{0}}{\sigma/\sqrt{n}}$ and $b=\frac{(n-1)S^{2}}{\sigma^{2}}$, which is one necessary condition of the ratio between $a$ and $\frac{b}{n-1}$ to be t-distributed.

Is the condition #2 an empirical rule to overwrite the theoretical conditions?

$\endgroup$
2

2 Answers 2

10
$\begingroup$
  1. This is the ideal case. When the population distribution is normal, then the sample mean is independent of the sample variance. Additionally, the numerator of the test statistic is a normal random variable while the denominator is a chi-square divided by its degrees of freedom; the definition of a t statistic. However, even when the population distribution is not normal the t test can be very useful. The question is "how far from normal is far enough for the t test to be ineffective".

  2. This is a commonly misunderstood claim. When $n>30$ the relative difference between a Student t and a normal distribution is sufficiently small so that someone decided they didn't have to print Student t tables past $\nu=30$ and could instead use normal tables with little consequence. Let me be clear: this is bad advice. Thirty is not a magic number and using this particular line in the sand as justification for using the t test is poor statistical practice.

$\endgroup$
12
  • 1
    $\begingroup$ Thank you Demetri for the answer. For your 2nd point, I think the misconception may have CLT to blame? Yes CLT dictates the sampling distribution is normal, but that is with known $\sigma$, which is almost never the case. $\endgroup$
    – lambda
    Commented Jan 25, 2022 at 4:07
  • 5
    $\begingroup$ The $t$-test is not robust to non-normality in general. The power of the test can be arbitrarily bad and the tail coverage for at least one of the two confidence limits can be arbitrarily inaccurate. Mild tail heaviness or moderate to severe skewness can really mess with the $t$-test. These problems are reasons that methods such as the bootstrap and the Wilcoxon test and its associated Hodges-Lehmann central tendency estimator exist. A good way to understand the severe limitations of the $t$-test is to understand that the SD can be an awful measure of dispersion. $\endgroup$ Commented Jan 25, 2022 at 14:14
  • 2
    $\begingroup$ Don't know of the best reference but studying Gini's mean difference will shed light on this. SD is not a good measure if the distribution is asymmetric. $\endgroup$ Commented Jan 25, 2022 at 19:22
  • 1
    $\begingroup$ @Demetri that page you link to only looks at quite mild deviations from normality. Of course everything works great then, because they avoided any cases where it didn't. $\endgroup$
    – Glen_b
    Commented Jan 27, 2022 at 4:50
  • 2
    $\begingroup$ ... On the other hand saying things like "moderately robust" (as I would) is arguably hardly any better, because while not obviously false, it offers no really useful guidance. However, that can be a deliberate strategy if one aims to lead people into actually investigating properties in a range of situations in the vicinity of their specific circumstances rather than relying on potentially quite misleading rules of thumb that make themselves out to be completely general. $\endgroup$
    – Glen_b
    Commented Jan 27, 2022 at 6:43
4
$\begingroup$

The significance level of the t-test is somewhat robust against mild to moderate deviations from normality, but we must take care not to claim too much.

By being overly general, the second recommendation (re $n>30$) is potentially dangerous.

Many posts on site discuss this or similar "sample size $30$" rules (I encourage you to search out some) but let me give an explicit, simple example that shows how easily we can strike a problem, since it seems this particular claim can be found even in high-level courses.

Angela and Bart are planning to use a pair of ordinary dice in a game, but because they're taking different roles in the game they have different concerns about possible bias. Chaina is an impartial person who will supervise the game for them.

Before the game, the players want to be confident that the dice are not badly biased against them, so they ask Chaina to roll the pair of dice 100 times to help them check the dice are not going to place either of them at a severe disadvantage. The outcome "double-six" (which is an important roll in the game) is of particular concern for them both; one benefits from it occurring, and the other from it not occurring.

Consequently Chaina rolls the two six-sided dice and records "1" every time they come up double-six, for each of 100 attempts. If the dice are fair and rolled properly, we should have to a good approximation a series of 100 independent Bernoulli trials (0/1 values), where the probability of 1 on each trial is 1/36.

After extensive investigation and copious application of slide rules, abacuses, calculators and counting on fingers and toes, I eventually managed to convince myself that the resulting sample of 0's and 1's meets the second condition in your post. That is, the sample size of 100 appears to exceed 30, by at least several more toes than I had in my possession. As an additional check, I asked my six year old neighbor for his thoughts on the matter, and he gladly lent his expert opinion at no charge, stating that 100 was indeed "lots" larger than 30, which more than made up for his unwillingness to contribute additional hands and feet to the task of computing relative sample sizes.

Consequently, according to this rule, we can proceed with our hypothesis tests with complete confidence.

Angela wishes to conduct a test of whether the mean - i.e. the proportion of recorded 1's (representing having got "double six") in 100 attempts is too low -- her alternative hypothesis -- compared to that for a pair of fair dice (100/36) at the $1\%$ level. She does a (one tailed, one sample) t-test which she believes should be fine, per your information.

Bart wishes to conduct a test of whether the mean - the proportion of 1's in 100 attempts is too high compared to that for a pair of fair dice (100/36) at the $5\%$ level. He does a (one tailed, one sample) t-test, which he feels assured is fine, again per your information.

Chaina also wants to check the dice are fair. She decides to follow the recommendations of a number of researchers in recent years, such as Benjamin et al. (2018)[1] and Ioannidis (2018)[2] to do a two-tailed one-sample t-test at the 0.005 ($\frac12\%$) level.

If the dice are fair (i.e. $H_0$ is true), what's the rejection rate (i.e. the true significance level, $\alpha$) for each of these three tests?

Try a simulation and see for yourself. Would any of them be satisfied with the rule you quote?

If our protagonists were to use R(*) to check the properties of their test, they will see approximately $6\%$ rejection rate on the $1\%$ test, roughly $0.07\%$ on the $5\%$ test and approximately $6\%$ on the two-tailed test that was to be conducted at $\frac12\%$.

How is that adequate?

Yet this is a pretty simple, "everyday" sort of situation.

If the "30" rule only applies sometimes, shouldn't they tell us when it does work? We seem to be missing important guidance here.

Clearly a more sophisticated rule is required; this one will not do.

Further, the common claim that the t-test is very robust obviously needs modification. It's not like we have some bizarre edge-case here. It's clearly not level-robust in this example --- and we haven't yet touched on its power behaviour, which is even more easily impacted.


* I made a point of this because it matters how your software treats a sample of all one value in the one sample t-test, which will happen occasionally. I agree with R's treatment of this situation; e.g. it gives a p-value of $0$ when the t-statistic has a non-zero numerator and a zero denominator in a two tailed test (where the hypothesized value is infinitely-many sample standard deviations from the sample mean)

Any potential software differences are in any case easily avoided by considering an analogous continuous distribution (which also avoids some other potential arguments).

Now our response is no longer a set of 0/1 outcomes but a continuous bimodal distribution with two very narrow peaks (i.e. where the spread around each mode is much smaller than the distance between the modes), with the lower-outcome mode having most of the probability (about 35/36 of it) and the higher-outcome peak having the remaining probability (1/36).

Imagine, perhaps, that we're recording (say) energy of individual photons where the energy is quantized (so we measure an extremely narrow range around a specific value for each of two values) and we're seeing one of the energies about 35 times as frequently as the other. Or imagine any other outcome that is bimodal with very narrow peaks and one much more frequent; other example circumstances that could fit this situation are not difficult to come up with.

With such a setup we will see similar results to those I present above, just from a simple continuous bimodal distribution.


Some example R code:

    pp = replicate(100000, t.test(rbinom(100, 1, 1/36), mu=1/36, 
                    alternative="less")$p.value)
    mean(pp<.01)   #  We need this to be close to 0.01
    pp = replicate(100000, t.test(rbinom(100, 1, 1/36), mu=1/36, 
                    alternative="greater")$p.value)
    mean(pp<.05)   #  We need this to be close to 0.05
    pp = replicate(100000, t.test(rbinom(100, 1, 1/36), 
                    mu=1/36)$p.value)
    mean(pp<.005)  #  We need this to be close to 0.005

To conduct the corresponding continuous data versions of the tests you could add a random (scaled and centered) symmetric beta value to each observation, with center 0 and range of something like (say) $10^{-4}$ times the distance between the modes.


[1] Benjamin, D.J., Berger, J., Johannesson, M., Nosek, B., Wagenmakers, E., Berk, R., et al., "Redefine statistical significance", Nat. Hum. Behav., 2018, no. 2, pp. 6–10.

[2] Ioannidis JPA., "The proposal to lower P-value thresholds to .005", JAMA. 2018; 319:1429–30.

$\endgroup$
4
  • $\begingroup$ Thank you @glen_b for the excellent counter example. I don’t know about the backgrounds of the discussion participants in this thread, but I do see everybody is stressing that we need to be cautious about over-optimistism of the normality violation. I don’t know if this is just a difference in industry vs. academia. I do see A/B testing practitioners in industry embrace the no-normality-is-ok more enthusiastically. Some books even say: no normality is OK, because the normality assumption is only on the sampling distribution of $\bar{Y}$,not on the ${Y_{i}}$. $\endgroup$
    – lambda
    Commented Jan 29, 2022 at 6:02
  • $\begingroup$ They say that a lot ... but you have to ask is that actually the case? (The answer is not if they want a t-statistic to actually have a t-distribution -- note that a t-statistic is not just a numerator. If you ignore that, at best we're dealing in approximations, and in that case how do we tell if it's a good approximation? We could do simulations like the above ... where we have already seen that sometimes the answer is that the approximation is very poor) $\endgroup$
    – Glen_b
    Commented Jan 29, 2022 at 6:12
  • 1
    $\begingroup$ ... So how do they know that they're not in one of those "the approximation very poor" situations? People weirdly write "academic" as if they imagined that academics never solve real problems when many do regularly. I have been an academic, but I work in industry and have done so in at least some capacity continually for more than 30 years. Being in industry does not mean you can't be careful, or thoughtful. It doesn't mean you can't make sure your claims hold water. It doesn't mean you can't do some actual checking that things have the properties you want them to. $\endgroup$
    – Glen_b
    Commented Jan 29, 2022 at 6:17
  • $\begingroup$ That’s true @glen_b; dividing the scenarios into industry and academia is too coarse. $\endgroup$
    – lambda
    Commented Jan 29, 2022 at 19:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.