# t-test on non normal data: type I/II error vs validity

First, I don't believe this is a duplicate post even though this topic has been brought up a million times. If it is, please point me to the relevant post and I will remove this one.

I am basically trying to understand what this post implicates in a practical sense: Independent samples t-test: Do data really need to be normally distributed for large sample sizes?

To summarize, the idea that one can apply the t-test to any data, regardless of sampling distribution, if the sample size is large enough is a contentious one. The argument for this claim (to my understanding, please correct if wrong) is

1. As n becomes larger, the numerator of the test statistic approaches a normal distribution by CLT. Although this is an asymptotic result, in practice this more or less holds on most datasets.
2. The denominator of the test statistic will probably not be anything close to a chi-sq. However, this is ok. By Slutsky's theorem, the ratio as a whole (i.e. test statistic) will approach a normal. For large n, even though the test statistic is asymptotically normal, we can still treat it as t-distributed because the t is very close to a standard normal when the sample size and thus df is large.
3. The answer shows a nice Monte Carlo simulation on log normal data which shows that there is no effect on type I or II errors.

In general, we want tests with high power conditioned on the type I error being controlled. However, type I error being controlled must also be conditioned on the test statistic being valid.

So my first question is: Is the idea of blindly using the t-test on any large dataset problematic not because of anything to do with type I or type II error but rather just the validity of test statistic? I don't mean the validity of the test statistic distribution. Rather whether the test stat can be interpreted to begin with. One comment in the post I linked states that the SD is not a good measure of dispersion of skewed data, but I'd like to get a second opinion on that in the context of test stat validity. Can I just tell someone that the test is invalid even though the test has good power and controlled FP? How would you convince them/show this is a problem in an applied setting?

My next question is: If the test statistic is valid, when does the t-test actually lose any practical power (or inflated type I error). This directly relates to a contradictory argument in the linked post which states a t-test loses significant power on skewed distributions. A simple way to answer this question would be to show a Monte Carlo simulation that "breaks" the t-test. However, I think is the very hard to do with large n. Can someone show an example where there is a practical loss of power or inflated type I error? Let's just say "practical" here is arbitrarily > 0.1. You can use anything you want' to break the t-test as long as the sample size is over 200 and the data has a mean and finite variance. Imbalanced groups, discrete data, ratio data, skew, etc are all fair game.

Thanks!

You are using the term "valid" in a binary manner, as in "either the t-test is valid or the t-test is invalid", which is problematic not only because there may be a grey area, but also because your definition of validity isn't really precise or objective ("can be interpreted", how could you distinguish whether it can or it can't?). Furthermore, you use it in a nonstandard manner. Most people would call a test "valid" if its type I and type II error behave well, i.e., it keep its level at least approximately, and should be unbiased, maybe with type II error somewhere close to the best you could achieve with the given information (which will not normally include knowledge of the underlying distribution). Also with this use of the term, the situation isn't that the test is either valid or invalid, as it's subjective and may depend on the application whether 4.47% is a good approximation to 5% or not, or whether anticonservativity as in 5.03% rejection rate can be tolerated.

Addressing your first question, I actually appreciate your care for the interpretability of the test statistic. I do think that it is important to interpret the test statistic, because if data are non-normal and the actual distribution is unspecified, the test statistic implicitly defines the alternative against which the test will be unbiased, namely all distributions for which the test statistic becomes too large in absolute value with a larger probability than the nominal level, i.e., the test statistic defines what is actually distinguished by the test. The t-test statistic in my view is fairly easy to interpret: Is the mean far away from the hypothesised one compared to the appropriately scaled sample standard deviation? The issue is not so much whether we can interpret it (surely we can), but rather whether this interpretation properly reflects what we are interested in in a given application. This may indeed not be the case in situations in which the mean or the sample standard deviation do not reflect the characteristics of the distribution well in which we are interested.

"Can I just tell someone that the test is invalid even though the test has good power and controlled FP? How would you convince them/show this is a problem in an applied setting?" The thing here is that (a) most people would use the term "valid" in such a way that it actually means "good power and controlled FP", so people wouldn't normally understand you. But also (b) note that power is not a fixed value but depends on the actual alternative against which you want to have good power. Now if the underlying distribution is non-normal, power against a normal alternative may not be what you're interested in. Maybe this is what you mean: Power against a normal alternative may be good, power against a non-normal alternative that is relevant in the given application may not be good. This will depend on what is actually relevant in your application, it needs to be specified for making a statement like this, and then it can be simulated. If this power with appropriately chosen non-normal alternative is good (we may be interested in a range of possible distributions really), I'd still think that the test should be called "valid".

In other words, my message is that your concern about validity can be translated into a concern regarding power against non-normal alternatives!

Note also that the definition of the test involves prominently both the test statistic and its distribution, as the latter determines when you reject, which is what a test really is about, so a test where the statistic is "valid" in your sense still requires an approximately correct distribution in order to work well, i.e., to be called "valid" by others.

Regarding your second question, the probably clearest example for such situations are situations with gross outliers. Assume a distribution $$P_{\mu,\sigma^2,\epsilon,x}=(1-\epsilon){\cal N}(\mu,\sigma^2)+\epsilon\delta_x$$, where $$\epsilon$$ is small and fixed, say 0.01, and $$\delta_x$$ is the Dirac/one-point distribution in $$x$$. $$x$$ can be interpreted as outlier if far away from $$\mu$$. The expected value of this is $$\mu^*=(1-\epsilon)\mu+\epsilon x$$, and the variance is, if I'm not mistaken, $$(1-\epsilon)((\mu-\mu^*)^2+\sigma^2)+\epsilon(x-\mu^*)^2=(1-\epsilon)(\epsilon(\mu-x)^2+\sigma^2).$$
If you consider $$x\to\infty$$, the numerator of the t-test statistic will be $$O_P(x)$$ and the denominator will be $$O_P(x^2)$$ (this is the essence of the example and will be true even if I got the variance slightly wrong because I did this quickly), meaning that for any arbitrarily large given $$n$$ you can find an $$x$$ that is so large that the denominator will dominate the numerator, or in other words, the absolute value of the t-statistic will be arbitrarily small with arbitrarily large probability, so that the test will almost never reject for any given value of $$\mu$$ (how large you have to choose the $$x$$ will depend on the $$\mu, n, \epsilon$$ for which you want to demonstrate that you can hardly ever reject). If you want to see a simulation, you should be able to make it up yourself from this description.

Addressing some of the issues mentioned earlier: "the idea that one can apply the t-test to any data" - I don't like the use of "can" here, as you can for sure compute the t-test for all kinds of data whether it's any good or not. And the quality, i.e., type I and type II error, depending on the actual alternatives considered, see above, is a gradual thing, not a binary one.

"in practice this more or less holds on most datasets" - this statement doesn't make much sense, as the theorem in question regards the assumed underlying distribution and $$n\to\infty$$, so it does not apply to a fixed dataset (note also that any theoretical probability distribution is a model and as such an idealisation; there isn't a well defined "true" underlying distribution given a dataset in reality).

"The denominator of the test statistic will probably not be anything close to a chi-sq. However, this is ok." It may or may not be OK (and actually this isn't binary either), it'll depend on the model you think is appropriate for the underlying process.

"Is the idea of blindly using the t-test on any large dataset problematic" - being blind is probably never a good idea; people recommend to check for skewness and outliers for a reason, and as shown before, outliers that lie sufficiently far out can destroy the power at arbitrarily large sample sizes. That said, I think that people who look at for example QQ-plots often tend to be overcritical regarding deviations from normality, as for large data the t-distribution of the test statistic is very often a good enough approximation for i.i.d. data; violations of independence are often far more critical (and the issue that significance for large data may occur too easily and one should really look at effect sizes and not just at p-values).

Note also that I have discussed whether to use a normal or a t-distribution for non-normal data here.