Justification for conducting independent samples t test on non normal data

I've conducted a study comparing two different teaching strategies in two different sections of the same class; I gave participants in both classes the same 5 tests and wanted to see if test grades in Class A differ significantly from test grades in class B.

My plan was to do an independent samples t test, but as it turns out, my data are non normal. I know that the independent samples t test is reasonably robust when samples depart from normality when each group has at least 25 participants, but unfortunately, one of the classes only has 20 participants. (The other has 33.)

As my goal is to compare the test means, I'd obviously prefer to do independent samples t tests over Mann-Whitney U tests. I've conducted both types of tests on the data and the Mann-Whitney U results follow the same pattern as the independent samples t test results – students took 5 tests total during the semester, and both independent samples t tests as well as Mann Whitney U tests were significant for the first 3 tests but not for the last 2 tests. I'd like to write up these results and submit them for publication, and, while I'll make a conceptual argument for using independent samples t tests based on the goal of my study (and will mention that the Mann Whitney U results followed the same pattern as the independent samples t tests), I'm wondering if anyone can recommend any articles I could cite to justify this.

• "As my goal is to compare the test means," ... then (assuming you don't have a suitable distributional model for the scores) why not a nonparametric test of means? Aug 4, 2022 at 2:30
• I do not think there will be a justification. While the test statistic you want (mean) is captured by a t-test, the assumptions are another thing, and need to hold if the test output is to be trusted. In this case, you might be able to use a permutation test with the difference in means as the test statistic.
– TJ27
Aug 4, 2022 at 13:34
• @Glen_b, I didn't think that there was a nonparametric test that assessed the mean. Happy that I'm wrong! Which one were you referring to? Aug 5, 2022 at 19:09
• The most obvious one is a straight permutation test of means; one can about as readily use the t-statistic itself as the statistic (which has some benefits); essentially all the nonparametric tests you have seen are permutation tests but mostly the ones you've seen are based on ranks (which is very useful if you lack a computer; something that hasn't really been a core issue for well over 40 years). For a basic description of such a test using the (absolute) difference of sample means as a statistic, see en.wikipedia.org/wiki/Permutation_test#Method ... ctd Aug 7, 2022 at 4:09
• ...a huge variety of convenient statistics can be used in other situations (many tests that you might wish were nonparametric can be), but if you're interested in means, that statistic or even a t-statistic (or its absolute value for a two tailed test) make sense. I do such tests regularly; in R it's typically a one line command to generate the possibilities (or more often, to sample from them) and another to compute the estimate of the p-value. It's typically less than a minute of additional effort over doing a t-test. Many posts on site discuss these tests and resampling tests more generally Aug 7, 2022 at 4:13

[In what follows, I will leave aside discussion of the suitability (or otherwise) of doing multiple two-sample t-tests (per your stated intention) against other options. If you wish to pursue those considerations further they might make for a good question.]

as it turns out, my data are non normal

You write this as if it were possible that the population distribution was normal, but of course it's not possible, since test scores are discrete and confined to an interval, whereas no normal distributions are either of those things.

Of course it might be possible that the test scores were approximately normal, but it sounds like you tested them. If you did, the hypothesis you tested was not "approximate normality" of the population. What you need to consider is not whether or not the population distribution of scores are actually normally distributed, but how much the potential amount and kind of non-normality in the population matters for the properties of your inference, which a test doesn't really tell you.

The t-test is reasonably level-robust (that is, the significance level is not very strongly impacted by many moderate deviations from normality) and the larger the sample, the less it is affected. That is, if the population distribution is not very skew (etc), the test will likely have about the right significance level.

I know that the independent samples t test is reasonably robust when samples depart from normality when each group has at least 25 participants

This kind of "hard number" is pretty much nonsense; it depends one how much of what kind of deviation from normality you have in your populations when H0 is true. Some non-normality that could be easily detected by eye (or by a test) would not badly impact your test at sample sizes below 20. Others could have a substantive impact when sample sizes are well into the hundreds. No single hard-and-fast number like that is reasonable without a clear indication of what circumstances it's meant to apply in -- it cannot be reasonable as a universal rule.

So you might well be okay, you have no great way to tell from the information you mention here.

If you have a good idea (outside of looking at your immediate samples) of roughly what the distribution shape should be expected to look like (if H0 were true), then you could investigate how much impact that might have on the significance level of a two sample t test.

A second concern would be power. Relative power is not going to substantially improve with larger samples, so issues like skewness and kurtosis may be more of a concern in that case.

"As my goal is to compare the test means,"

I see several possibilities besides the sort of approach discussed above.

- if you have some basis to assume a plausible model distribution under H0 and under H1 (as discussed above), it would be possible instead to obtain a more suitable test of means under that assumption. I presume that you don't have such a distributional model.

- you could add some suitable assumptions that would make a rank-based test a test of means (such as a pure-shift alternative); the Mann-Whitney has good power near the normal in such a case. The main problem comes when the added assumption is not suitable. For example it's possible for the Mann-Whitney to reject in the opposite direction to the direction that the means differ in.

- you could perform a nonparametric test of means

Let us explore this option.

There are two common choices, both resampling tests (though not the only possibilities):

1. A bootstrap test of means

2. A permutation test of means

The permutation test will avoid exceeding your desired significance level (presuming other assumptions still hold) but require an assumption of exchangeability when H0 is true. In this case that implies that not only should the means be the same when H0 is true, the population distribution shape should be the same (that when H0 is true, the distributions are the same). The shape and even spread can differ when H0 is not the case, so the current data won't necessarily help you decide if this assumption makes sense -- it's about what the situation would be in the populations under H0 (not that you should look at the data to choose what you are testing).

This is a much weaker assumption than you need under the usual equal-variance t-test but it still assumes something.

The bootstrap doesn't necessarily require you to make that assumption under the null, but you no longer have the strong control over type I error; the bootstrap works better with moderately large samples.

The two procedures are somewhat similar in general approach, but for the moment I'll just describe a permutation test of means.

The basic idea is that under $$H_0$$, since the distributions will be the same when their means are the same, the group labels are arbitrary when $$H_0$$ is true; a particular observation could have come from either group.

Consequently, when $$H_0$$ is true we can interchange the group labels on the observations without changing either distribution. So we proceed to do exactly that. Ideally we consider every possible rearrangement of labels, of which (under $$H_0$$) our particular should be a 'random' member.

We choose a test statistic that will tend to behave differently when $$H_0$$ is false than when it is true. If we compute that statistic on every one of our rearranged samples, we have the permutation distribution of the test statistic under $$H_0$$. We then can designate some proportion $$\alpha$$ of those - the ones most consistent with our alternative - as being in our rejection region. These are the cases where the test statistic most strongly suggests the alternative, and so they should occur more often than a fraction $$\alpha$$ of the time when $$H_0$$ is false.

I presume you want a two tailed test, in which case you could for example use as your test statistic the absolute difference in the sample means $$T = |\bar{x}_1-\bar{x}_2|$$.

So already we have a way to get a reject/don't reject decision.

We can also compute a p-value. It's the largest choice of $$\alpha$$ for which we'd reject $$H_0$$; equivalently it's the fraction of the distribution that is at least as extreme as the sample statistic. With the specific $$T$$ we defined, we calculate the proportion of the permutation distribution of $$T$$ values at least as large as the $$T$$ calculated on the sample (note that this includes the one calculated on the sample itself; it's part of that proportion).

The problem with this is that when sample sizes are not small, we have extremely large numbers of possible rearrangements of sample labels; the numbers very quickly surpass the trillions (as they will in your case). Clever algorithms can reduce the computations you need to do to get a p-value but in any case we can approximate the p-value as accurately as we desire by randomly sampling the distribution of permutations rather than evaluating all of them.

The approach is then to repeat many times (many thousands at least), random reallocation of the group labels (i.e. randomly choose which observations count as group 1 and put the rest in group 2), and compute $$T$$ for that random exchange of labels. Keep all the $$T$$ values, add in your original sample's $$T$$ value, and then figure out the proportion of them that are at least as large as that one (in practice we don't explicitly extend the list, we can just add 1 to the count in the numerator and denominator, since $$T$$ is always as large as itself).

Typically with sample sizes like this we could do hundreds of thousands of them in a few seconds to a few tens of seconds - and I recommend doing a hundred thousand or so, in order that you get a fairly precise bound on the $$p$$ value (I would want a couple of figures of accuracy on a p-value near 5%, so that suggests that the standard error should ideally be smaller than 0.0005 implying at least 190,000 resamples. Typically I settle for 100,000. Many people do far fewer, but given the run time is often less than the roughly a half-minute to a minute that it takes to set up I don't mind waiting for 10 or 20 seconds at all.

[Wikipedia also describes this test here]

As mentioned previously, one can about as readily use the t-statistic itself as the statistic instead. One of the benefits is that the t-statistic is still interpretable as a number of standard errors that the means are apart.

You can use many other statistics with permutation tests -- many simple tests that you might wish were distribution-free can be made so with this approach. Many more will be approximately distribution free, and bootstrapping can do so with a much wider variety of statistics again.

Example done in R. This reads in a pair of samples whose values I made up (of the same size as the samples you mentioned, 20 and 33) and computes $$T$$ on 100,000 relabellings of the observed data.

# read in and set up data
x1 = scan()
80 86 47 91 58 29 51 62 69 38 56 60 81 50 38
22 78 48 27 45

x2 = scan()
55 75 36 57 79 66 74 50 72 58 71 60 50 56 32
86 56 99 72 67 80 75 43 60 43 53 53 47 66 72
61 61 71

n1 = length(x1) # find sample sizes
n2 = length(x2)
T0 = abs(mean(x1)-mean(x2)) # calculate T on the sample
xall = c(x1,x2) # combine samples
tot = sum(xall)

nsim = 100000
T = replicate(nsim,{
m1=mean(sample(xall,n1))
m2=(tot-n1*m1)/n2
abs(m1-m2)
})
gecount = sum(T>=T0)
p = (gecount+1)/(nsim+1)
p


which produces the output:

> p
[1] 0.1803282


repeating the simulation

> p
[1] 0.1801582


so we can indeed happily say that $$p\approx 0.180$$ and that in this case we should not reject the null hypothesis.

Note that there are two typical bases for inference; random sampling of the population of interest and random allocation to treatment. This approach should be suitable in either case, but you would slightly change how you write your conclusion so suit.

This code ran in about 5 seconds on the weakest laptop in the house. I also pasted the code into the code window at https://rdrr.io/snippets which again took about 5 seconds. (Though I had to retry a couple of times; it sometimes says "Service unavailable", I presume from too many people using it at once.)

I have made my code a bit more 'wordy' than I'd usually write for a quick test, which hopefully help with seeing what's going on. One thing that might not be immediately clear is the computation of $$T$$; note that because the sum of both samples is always constant, we can work out the mean of the second sample by knowing the mean of the first sample; the calculation of $$T$$ inside replicate takes advantage of that.

responses to comments (though much of it ends up recapitulating the discussion above):

One question...when you say "What you need to consider is not whether or not the population distribution of scores are actually normally distributed, but how much the potential amount and kind of non-normality in the population matters for the properties of your inference, which a test doesn't really tell you," what exactly do you mean by that? –

It's a little difficult to tell what aspect of it is unclear. As addressed above, the distribution of your variable certainly cannot be normal, but this is not of itself particularly important.

Nothing is normal in practice. Normality is a model, an abstraction, a convenient approximation, like a physicist's spherical cow.

For some purposes, the spherical cow is a fine approximation, yielding useful* answers; for other purposes it may not be helpful at all. It's the same with a normal distribution; for some things, even a very rough approximation by a normal distribution may be sufficient for some particular purpose, for others it may not do at all.

* as George Box, put it: "Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful"

I'm evaluating the effectiveness of a new teaching strategy - one class recieved the strategy and one did not. My control group's data is normally distributed,

No, it isn't. As discussed above, the variable is discrete and bounded between 0 and some fixed maximum, right?

I presume you mean instead that you tested normality and didn't reject. You have simply committed a type II error, because $$H_0$$ is certainly false. At best a hypothesis test could tell you what is already obvious with no data at all and at worse - as here - it could fool you into asserting an obvious falsehood.

Can it tell you something useful? e.g. can it tell you when the normal approximation will yield useful answers? not consistently. In large samples you'll reject normality, even if it had no consequences at all for your conclusions. In small samples you'll typically fail to reject normality, even if the potential consequences are substantial. As sample size increases, a test for normality tends to get more sensitive - picking up ever smaller effects - when in many cases a given effect size matters less and less.

but my experimental group's data is moderately negatively skewed, which is what I was theorizing would happen as I suspected the teaching strategy would lead to higher test scores. So the non-normality was expected. Is that at all what you mean by saying "how much the potential amount and kind of non-normality in the population matters for the properties of your inference"? –

No, I was referring to its impact on the properties of the test (or whatever other analysis you might have done) - in the case of a test specifically, the actual significance level ($$\alpha$$) and power.

Under $$H_0$$ (where if the treatment had no effect you would presumably expect the experimental group to behave like the control group), an anticipated skewness under $$H_1$$ is immaterial to significance level. With a bounded variable, if you expect a roughly hill shaped /unimodal not-very skewed distribution for both conditions under $$H_0$$, your significance level is probably fine, and power for smallish effect sizes (i.e. where you tend to want it most) is probably also just fine.

I'd caution against using the same data you want to use in the test to choose your tests $$-$$ and hence your hypotheses.

• a very hearty belated thanks for your detailed response! Can't tell you how much I appreciate it. One question...when you say "What you need to consider is not whether or not the population distribution of scores are actually normally distributed, but how much the potential amount and kind of non-normality in the population matters for the properties of your inference, which a test doesn't really tell you," what exactly do you mean by that? Mar 15, 2023 at 3:06
• I'm evaluating the effectiveness of a new teaching strategy - one class recieved the strategy and one did not. My control group's data is normally distributed, but my experimental group's data is moderately negatively skewed, which is what I was theorizing would happen as I suspected the teaching strategy would lead to higher test scores. So the non-normality was expected. Is that at all what you mean by saying "how much the potential amount and kind of non-normality in the population matters for the properties of your inference"? Mar 15, 2023 at 3:15
• That's rather a lot to respond to as a comment, though answers to much of it can be found by using the search function. I'll make some brief responses in an edit to my answer. Mar 16, 2023 at 0:14
• I'm happy to clarify any of those points further, or to take it up in chat. Mar 16, 2023 at 0:37

I will echo the other answer here and say the comparison of several t-tests is not a good practice (Brown, 1990). In any case, I wanted to provide a shorter alternative answer. One could simply use a Welch's t-test without really caring much about the distributional attributes associated with your data. Unlike the t-test, the Welch t-test is defined so:

$$t = \frac{\bar{X_1}-\bar{X_2}}{\sqrt{\frac{s^2_1}{n_1}+\frac{s^2_2}{n_2}}}$$

where $$\bar{X}$$ is the mean of a sample, $$s^2$$ is the variance of a sample, and $$n$$ is the sample size. The reason why the Welch t-test is different is because it doesn't unnecessarily pool together the variance like a Student t-test. This means you don't have to concern yourself with the equality of variance between each group. That said, some have remarked that for non-parametric t-tests, one should carefully select between the Mann-Whitney U-test and the Welch t-test. However, some simulations have shown that Welch pretty much does the trick in all cases with only minor losses in power in extreme cases (Delacre et al., 2017). So from extreme departures of normality, the Welch basically covers your bases and you don't really have to concern yourself with CLT definitions of the test to begin with.

The Welch is also fairly easy to accomplish using R. Here I quickly simulate some data and run the t.test function, which by default uses the Welch version:

#### Sim Data ####
group <- factor(
rbinom(n=1000,size=1,prob=.5),
labels = c("Ctrl","Trt")
)
outcome <- rnorm(n=1000)

#### Welch ####
t.test(outcome ~ group)


Results shown below:

Welch Two Sample t-test

data:  outcome by group
t = -1.2624, df = 993.49, p-value = 0.2071
alternative hypothesis: true difference in means between group Ctrl and group Trt is not equal to 0
95 percent confidence interval:
-0.2105298  0.0456953
sample estimates:
mean in group Ctrl  mean in group Trt
-0.06989047         0.01252676


Leaving the arguments of normality aside, I think using the Welch by default is generally good practice because it simply negates a lot of the issues associated with using the standard Student t-test.

Citations

• Brown, J. D. (1990). The use of multiple t-tests in language research. TESOL Quarterly, 24(4), 770–773. https://doi.org/10.2307/3587135
• Delacre, M., Lakens, D., & Leys, C. (2017). Why psychologists should by default use Welch’s t-test instead of Student’s t-test. International Review of Social Psychology, 30(1), 92. https://doi.org/10.5334/irsp.82