I have used paired samples t-tests to compare 3x time-point measurements, A/ B/ C.

A and B t(102)=4.988 p=.000 A and C t(102)=4.939 p=.000 B and C t(102)=.346 p=.730

A, B, and C were all significant on the Kolmogorov-Smirnov, so none were normally distributed.

My understanding is that this is not a problem due to the t-test being robust to normality violations when sample size is above 20.

My concern is that these significance results are erroneous due to not having corrected the alpha level via Bonferroni to accommodate the extra (one) comparison, to avoid a higher than 5% chance of a Type I error. Am I wrong? Does the strong p value get me off the hook? I have already written up and submitted my dissertation :(

Maybe a Friedman's should have been used.

**In response to Glen_b Thanks for your response. I did the dissertation 2 years ago so I'm trying to remember why I didn't do a correction. Would I be correct in saying, a Bonferroni adjustment in this case would be: alpha level divided by number of tests. Therefore is that .05/2=.025? and in that case, even if I were to apply a Bonferroni adjustment, wouldn't my significant results still stand? Therefore making the Bonferroni redundant? I have stated in the text I wanted to avoid Type II error because I wanted to increase the chance of a significant finding (now that I read it it sounds outrageous!).. I had read a paper by Armstrong (2014) who cited Perneger's (1998) six reasons for not doing the Bonferroni. Doesn't it feel illogical to think that doing two separate isolated tests have anything to do with eachother. Isn't it like spinning a roulette wheel twice, in that it starts from scratch each time!


2 Answers 2


What matters for correctness of the null distribution (and hence, type I error rates) is
(a) the distribution of pair-differences
(b) when $H_0$ is true

Not the raw values, nor even the sample differences when $H_0$ is likely false.

There's no specific sample size at which you can say "it's definitely fine now, the per-test $\alpha$ will be what I chose it to be", but if you have skewed/heavy tailed variables you're typically going to see a lowering of significance level rather than an increase, so if $\alpha>0.05$ is your main concern it's probably not a big deal on that score. You might have more of an issue with power in that situation, though.

Corrections for multiple testing is a separate issue to the concern about the shape of the distribution. One doesn't fix the other.

One question you're likely to get from people is why would you use three t-tests for this rather than one omnibus test and post hoc comparisons?

I'm not saying that this choice was wrong, necessarily, but it isn't the most common analysis and you can be pretty sure it will come up.

  • $\begingroup$ Thanks for your response, I've responded by appending my original question which hopefully adds more important information. By the way would it make a difference were the variables A, B, and C to only have 4 possible scores? $\endgroup$ Commented Feb 7, 2023 at 23:54
  • 1
    $\begingroup$ You tuck a big question in at the end of your comment as if it's almost nothing. It's worth worrying about as a whole question of its own (though questions like it have been addressed many times here). If there's only 4 possible scores obviously the variable is not-normal, there's absolutely no point testing a hypothesis that is certainly false. Can you say something about these scores? What are they measuring? $\endgroup$
    – Glen_b
    Commented Feb 8, 2023 at 0:41
  • $\begingroup$ Given the two year gap most of my parenthetical comment at the end was irrelevant, I've removed it. Edit: If you're adjusting for multiple testing using Bonferroni, you conventionally divide the significance level by the number of tests, but didn't you do three comparisons, not two? It sounds like you're now focused on a different issue than I was responding to (that of whether you should correct for multiple tests at all). Note that a number of posts on site discuss issues related to multiple-testing corrections, some of those may be relevant. You might want to consider a new post perhaps. $\endgroup$
    – Glen_b
    Commented Feb 8, 2023 at 0:46
  • $\begingroup$ I see four distinct issues you're asking about now: (i) what you assume in a paired t-test for the derivation of the null distribution of the test statistic and how robust the adherence to the significance level is to it; (ii) the mechanics of how to do a Bonferroni (or whatever other multiple testing adjustment you might consider); (iii) whether you actually need to do multiple testing correction at all; (iv) what analysis to use with a response consisting of four possible values (though it's not yet clear if this is ordinal or interval or something else). These could each be a question $\endgroup$
    – Glen_b
    Commented Feb 8, 2023 at 0:59
  • $\begingroup$ Thanks Glen - if each variable is only compared twice, doesn't that make two comparisons? e.g. A vs B and A vs C etc. The variables are interval, they represent remembering an item or not, scores from 0-4. Are you saying t-test is inappropriate because it compares two normal distributions against eachother. I am crying because damn statistician friend advised me to use t-tests at the time, despite the low ranges. Are the findings valid is the only question. When I graph the data, it visually corresponds with the significance findings. $\endgroup$ Commented Feb 8, 2023 at 16:31

If the null hypothesis is vaguely stated that "there is no difference after timepoint A", then B and C taken together represent a sparse summary of the post-baseline timeseries. If there truly were no difference, you would inflate the false positive error rate by comparing B to A and then C to A, so a correction would be needed. But we might expect B and C to be very correlated, so the Bonferroni would be too conservative. Other multiplicity corrections like a Benjamini-Hochberg correction could recover some lost $\alpha$.

In this scenario, I would recommend linear modeling with ANCOVA where you could fit the model given by:

$$ E[Y_t | t; t>0] = \beta_0 + \beta_1 Y_0 + \beta_2 t$$

and the null hypothesis would be $$\mathcal{H}_0: \beta_0 = \beta_2 = 0$$

This would be a powerful modeling approach because it is conducted by a single (2 degree of freedom) test, and the sensitivity analysis reduces to analyzing residuals that have been adjusted for obvious exogenous factors, such as baseline and time. If the design is balanced, there's no need to account for blocking by subject, but more generally you can fit the model by GEE using exchangeable correlation structure - this has nice connections to the $t$-test with unequal variance assumptions. In fact, if we omit the "time" term $\beta_2$ and constrain $\beta_1=1$ by an offset, this is exactly the paired t-test, so our ANCOVA approach is arguably more general.

As @GlenB points out, the non-normality of residuals may continue to affect inference, but no general rule of thumb is acceptable to card-carrying statisticians - extensive inspection by simulations and sensitivity analyses would be needed to definitively say that the test doesn't have adequate power, or is of the incorrect size.

  • $\begingroup$ Thanks, would ANCOVA handle the non-normality? $\endgroup$ Commented Feb 8, 2023 at 19:01
  • $\begingroup$ @matthew only slightly better than a t-test for the reasons I mention, but the point GlenB and I are making is that you have every reason to use it anyway when your question has to do with mean differences. We expect that the test is valid and better power against alternatives inspite of everything. Practically it may be hard to convince reviewer 2, but testing nonnormality is nearly universally agreed upon as misleading among statsfolk. $\endgroup$
    – AdamO
    Commented Feb 8, 2023 at 19:30
  • $\begingroup$ Ok, thanks, you mean using K-S and Shapiro-Wilk etc is misleading? That's great to hear the t-test was the right choice here, it is just to prepare a satisfactory defence of it which may be problematic :) $\endgroup$ Commented Feb 8, 2023 at 19:45
  • $\begingroup$ @Matthew K-S tests the homogeneity of two distributions, by virtue of the fact we expect a shift in post-baseline values, K-S is useless here. Shapiro Wilk is formally a test of normality, and yes I claim it's useless - since you have a small sample it's underpowered! The most powerful "test" of normality is your eyes. Don't forget to consider that the log transform is often appropriate for skewed and positive data. $\endgroup$
    – AdamO
    Commented Feb 8, 2023 at 20:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.