In a muddle with Paired Samples t-tests on pre vs during vs post I have used paired samples t-tests to compare 3x time-point measurements, A/ B/ C.
A and B t(102)=4.988 p=.000
A and C t(102)=4.939 p=.000
B and C t(102)=.346  p=.730
A, B, and C were all significant on the Kolmogorov-Smirnov, so none were normally distributed.
My understanding is that this is not a problem due to the t-test being robust to normality violations when sample size is above 20.
My concern is that these significance results are erroneous due to not having corrected the alpha level via Bonferroni to accommodate the extra (one) comparison, to avoid a higher than 5% chance of a Type I error. Am I wrong?
Does the strong p value get me off the hook? I have already written up and submitted my dissertation :(
Maybe a Friedman's should have been used.
**In response to Glen_b
Thanks for your response. I did the dissertation 2 years ago so I'm trying to remember why I didn't do a correction. Would I be correct in saying, a Bonferroni adjustment in this case would be: alpha level divided by number of tests. Therefore is that .05/2=.025? and in that case, even if I were to apply a Bonferroni adjustment, wouldn't my significant results still stand? Therefore making the Bonferroni redundant?
I have stated in the text I wanted to avoid Type II error because I wanted to increase the chance of a significant finding (now that I read it it sounds outrageous!).. I had read a paper by Armstrong (2014) who cited Perneger's (1998) six reasons for not doing the Bonferroni.
Doesn't it feel illogical to think that doing two separate isolated tests have anything to do with eachother. Isn't it like spinning a roulette wheel twice, in that it starts from scratch each time!
 A: What matters for correctness of the null distribution (and hence, type I error rates) is
(a) the distribution of pair-differences
(b) when $H_0$ is true
Not the raw values, nor even the sample differences when $H_0$ is likely false.
There's no specific sample size at which you can say "it's definitely fine now, the per-test $\alpha$ will be what I chose it to be", but if you have skewed/heavy tailed variables you're typically going to see a lowering of significance level rather than an increase, so if $\alpha>0.05$ is your main concern it's probably not a big deal on that score. You might have more of an issue with power in that situation, though.
Corrections for multiple testing is a separate issue to the concern about the shape of the distribution. One doesn't fix the other.
One question you're likely to get from people is why would you use three t-tests for this rather than one omnibus test and post hoc comparisons?
I'm not saying that this choice was wrong, necessarily, but it isn't the most common analysis and you can be pretty sure it will come up.
A: If the null hypothesis is vaguely stated that "there is no difference after timepoint A", then B and C taken together represent a sparse summary of the post-baseline timeseries. If there truly were no difference, you would inflate the false positive error rate by comparing B to A and then C to A, so a correction would be needed. But we might expect B and C to be very correlated, so the Bonferroni would be too conservative. Other multiplicity corrections like a Benjamini-Hochberg correction could recover some lost $\alpha$.
In this scenario, I would recommend linear modeling with ANCOVA where you could fit the model given by:
$$ E[Y_t | t; t>0] = \beta_0 + \beta_1 Y_0 + \beta_2 t$$
and the null hypothesis would be $$\mathcal{H}_0: \beta_0 = \beta_2 = 0$$
This would be a powerful modeling approach because it is conducted by a single (2 degree of freedom) test, and the sensitivity analysis reduces to analyzing residuals that have been adjusted for obvious exogenous factors, such as baseline and time. If the design is balanced, there's no need to account for blocking by subject, but more generally you can fit the model by GEE using exchangeable correlation structure - this has nice connections to the $t$-test with unequal variance assumptions. In fact, if we omit the "time" term $\beta_2$ and constrain $\beta_1=1$ by an offset, this is exactly the paired t-test, so our ANCOVA approach is arguably more general.
As @GlenB points out, the non-normality of residuals may continue to affect inference, but no general rule of thumb is acceptable to card-carrying statisticians - extensive inspection by simulations and sensitivity analyses would be needed to definitively say that the test doesn't have adequate power, or is of the incorrect size.
