[In what follows, I will leave aside discussion of the suitability (or otherwise) of doing multiple two-sample t-tests (per your stated intention) against other options. If you wish to pursue those considerations further they might make for a good question.]
as it turns out, my data are non normal
You write this as if it were possible that the population distribution was normal, but of course it's not possible, since test scores are discrete and confined to an interval, whereas no normal distributions are either of those things.
Of course it might be possible that the test scores were approximately normal, but it sounds like you tested them. If you did, the hypothesis you tested was not "approximate normality" of the population. What you need to consider is not whether or not the population distribution of scores are actually normally distributed, but how much the potential amount and kind of non-normality in the population matters for the properties of your inference, which a test doesn't really tell you.
The t-test is reasonably level-robust (that is, the significance level is not very strongly impacted by many moderate deviations from normality) and the larger the sample, the less it is affected. That is, if the population distribution is not very skew (etc), the test will likely have about the right significance level.
I know that the independent samples t test is reasonably robust when samples depart from normality when each group has at least 25 participants
This kind of "hard number" is pretty much nonsense; it depends one how much of what kind of deviation from normality you have in your populations when H0 is true. Some non-normality that could be easily detected by eye (or by a test) would not badly impact your test at sample sizes below 20. Others could have a substantive impact when sample sizes are well into the hundreds. No single hard-and-fast number like that is reasonable without a clear indication of what circumstances it's meant to apply in -- it cannot be reasonable as a universal rule.
So you might well be okay, you have no great way to tell from the information you mention here.
If you have a good idea (outside of looking at your immediate samples) of roughly what the distribution shape should be expected to look like (if H0 were true), then you could investigate how much impact that might have on the significance level of a two sample t test.
A second concern would be power. Relative power is not going to substantially improve with larger samples, so issues like skewness and kurtosis may be more of a concern in that case.
"As my goal is to compare the test means,"
I see several possibilities besides the sort of approach discussed above.
- if you have some basis to assume a plausible model distribution under H0 and under H1 (as discussed above), it would be possible instead to obtain a more suitable test of means under that assumption. I presume that you don't have such a distributional model.
- you could add some suitable assumptions that would make a rank-based test a test of means (such as a pure-shift alternative); the Mann-Whitney has good power near the normal in such a case. The main problem comes when the added assumption is not suitable. For example it's possible for the Mann-Whitney to reject in the opposite direction to the direction that the means differ in.
- you could perform a nonparametric test of means
Let us explore this option.
There are two common choices, both resampling tests (though not the only possibilities):
A bootstrap test of means
A permutation test of means
Both have their advantages and disadvantages.
The permutation test will avoid exceeding your desired significance level (presuming other assumptions still hold) but require an assumption of exchangeability when H0 is true. In this case that implies that not only should the means be the same when H0 is true, the population distribution shape should be the same (that when H0 is true, the distributions are the same). The shape and even spread can differ when H0 is not the case, so the current data won't necessarily help you decide if this assumption makes sense -- it's about what the situation would be in the populations under H0 (not that you should look at the data to choose what you are testing).
This is a much weaker assumption than you need under the usual equal-variance t-test but it still assumes something.
The bootstrap doesn't necessarily require you to make that assumption under the null, but you no longer have the strong control over type I error; the bootstrap works better with moderately large samples.
The two procedures are somewhat similar in general approach, but for the moment I'll just describe a permutation test of means.
The basic idea is that under $H_0$, since the distributions will be the same when their means are the same, the group labels are arbitrary when $H_0$ is true; a particular observation could have come from either group.
Consequently, when $H_0$ is true we can interchange the group labels on the observations without changing either distribution. So we proceed to do exactly that. Ideally we consider every possible rearrangement of labels, of which (under $H_0$) our particular should be a 'random' member.
We choose a test statistic that will tend to behave differently when $H_0$ is false than when it is true. If we compute that statistic on every one of our rearranged samples, we have the permutation distribution of the test statistic under $H_0$. We then can designate some proportion $\alpha$ of those - the ones most consistent with our alternative - as being in our rejection region. These are the cases where the test statistic most strongly suggests the alternative, and so they should occur more often than a fraction $\alpha$ of the time when $H_0$ is false.
I presume you want a two tailed test, in which case you could for example use as your test statistic the absolute difference in the sample means $T = |\bar{x}_1-\bar{x}_2|$.
So already we have a way to get a reject/don't reject decision.
We can also compute a p-value. It's the largest choice of $\alpha$ for which we'd reject $H_0$; equivalently it's the fraction of the distribution that is at least as extreme as the sample statistic. With the specific $T$ we defined, we calculate the proportion of the permutation distribution of $T$ values at least as large as the $T$ calculated on the sample (note that this includes the one calculated on the sample itself; it's part of that proportion).
The problem with this is that when sample sizes are not small, we have extremely large numbers of possible rearrangements of sample labels; the numbers very quickly surpass the trillions (as they will in your case). Clever algorithms can reduce the computations you need to do to get a p-value but in any case we can approximate the p-value as accurately as we desire by randomly sampling the distribution of permutations rather than evaluating all of them.
The approach is then to repeat many times (many thousands at least), random reallocation of the group labels (i.e. randomly choose which observations count as group 1 and put the rest in group 2), and compute $T$ for that random exchange of labels. Keep all the $T$ values, add in your original sample's $T$ value, and then figure out the proportion of them that are at least as large as that one (in practice we don't explicitly extend the list, we can just add 1 to the count in the numerator and denominator, since $T$ is always as large as itself).
Typically with sample sizes like this we could do hundreds of thousands of them in a few seconds to a few tens of seconds - and I recommend doing a hundred thousand or so, in order that you get a fairly precise bound on the $p$ value (I would want a couple of figures of accuracy on a p-value near 5%, so that suggests that the standard error should ideally be smaller than 0.0005 implying at least 190,000 resamples. Typically I settle for 100,000. Many people do far fewer, but given the run time is often less than the roughly a half-minute to a minute that it takes to set up I don't mind waiting for 10 or 20 seconds at all.
[Wikipedia also describes this test here]
As mentioned previously, one can about as readily use the t-statistic itself as the statistic instead. One of the benefits is that the t-statistic is still interpretable as a number of standard errors that the means are apart.
You can use many other statistics with permutation tests -- many simple tests that you might wish were distribution-free can be made so with this approach. Many more will be approximately distribution free, and bootstrapping can do so with a much wider variety of statistics again.
Example done in R. This reads in a pair of samples whose values I made up (of the same size as the samples you mentioned, 20 and 33) and computes $T$ on 100,000 relabellings of the observed data.
# read in and set up data
x1 = scan()
80 86 47 91 58 29 51 62 69 38 56 60 81 50 38
22 78 48 27 45
x2 = scan()
55 75 36 57 79 66 74 50 72 58 71 60 50 56 32
86 56 99 72 67 80 75 43 60 43 53 53 47 66 72
61 61 71
n1 = length(x1) # find sample sizes
n2 = length(x2)
T0 = abs(mean(x1)-mean(x2)) # calculate T on the sample
xall = c(x1,x2) # combine samples
tot = sum(xall)
nsim = 100000
T = replicate(nsim,{
m1=mean(sample(xall,n1))
m2=(tot-n1*m1)/n2
abs(m1-m2)
})
gecount = sum(T>=T0)
p = (gecount+1)/(nsim+1)
p
which produces the output:
> p
[1] 0.1803282
repeating the simulation
> p
[1] 0.1801582
so we can indeed happily say that $p\approx 0.180$ and that in this case we should not reject the null hypothesis.
Note that there are two typical bases for inference; random sampling of the population of interest and random allocation to treatment. This approach should be suitable in either case, but you would slightly change how you write your conclusion so suit.
This code ran in about 5 seconds on the weakest laptop in the house. I also pasted the code into the code window at https://rdrr.io/snippets which again took about 5 seconds. (Though I had to retry a couple of times; it sometimes says "Service unavailable", I presume from too many people using it at once.)
I have made my code a bit more 'wordy' than I'd usually write for a quick test, which hopefully help with seeing what's going on. One thing that might not be immediately clear is the computation of $T$; note that because the sum of both samples is always constant, we can work out the mean of the second sample by knowing the mean of the first sample; the calculation of $T$ inside replicate takes advantage of that.
responses to comments (though much of it ends up recapitulating the discussion above):
One question...when you say "What you need to consider is not whether or not the population distribution of scores are actually normally distributed, but how much the potential amount and kind of non-normality in the population matters for the properties of your inference, which a test doesn't really tell you," what exactly do you mean by that? –
It's a little difficult to tell what aspect of it is unclear. As addressed above, the distribution of your variable certainly cannot be normal, but this is not of itself particularly important.
Nothing is normal in practice. Normality is a model, an abstraction, a convenient approximation, like a physicist's spherical cow.
For some purposes, the spherical cow is a fine approximation, yielding useful* answers; for other purposes it may not be helpful at all. It's the same with a normal distribution; for some things, even a very rough approximation by a normal distribution may be sufficient for some particular purpose, for others it may not do at all.
* as George Box, put it: "Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful"
I'm evaluating the effectiveness of a new teaching strategy - one class recieved the strategy and one did not. My control group's data is normally distributed,
No, it isn't. As discussed above, the variable is discrete and bounded between 0 and some fixed maximum, right?
I presume you mean instead that you tested normality and didn't reject. You have simply committed a type II error, because $H_0$ is certainly false. At best a hypothesis test could tell you what is already obvious with no data at all and at worse - as here - it could fool you into asserting an obvious falsehood.
Can it tell you something useful? e.g. can it tell you when the normal approximation will yield useful answers? not consistently. In large samples you'll reject normality, even if it had no consequences at all for your conclusions. In small samples you'll typically fail to reject normality, even if the potential consequences are substantial. As sample size increases, a test for normality tends to get more sensitive - picking up ever smaller effects - when in many cases a given effect size matters less and less.
but my experimental group's data is moderately negatively skewed, which is what I was theorizing would happen as I suspected the teaching strategy would lead to higher test scores. So the non-normality was expected. Is that at all what you mean by saying "how much the potential amount and kind of non-normality in the population matters for the properties of your inference"? –
No, I was referring to its impact on the properties of the test (or whatever other analysis you might have done) - in the case of a test specifically, the actual significance level ($\alpha$) and power.
Under $H_0$ (where if the treatment had no effect you would presumably expect the experimental group to behave like the control group), an anticipated skewness under $H_1$ is immaterial to significance level. With a bounded variable, if you expect a roughly hill shaped /unimodal not-very skewed distribution for both conditions under $H_0$, your significance level is probably fine, and power for smallish effect sizes (i.e. where you tend to want it most) is probably also just fine.
I'd caution against using the same data you want to use in the test to choose your tests $-$ and hence your hypotheses.
