Are the results of my paired tests false positive?

Question

I am dealing with a strange situation which I hope you can come up with a solution. I have 3 pared group(n=7 for each) which I tried to see if they are significantly difference and if so, in which pairs. The means for Groups (G1-3) are 33.01, 31.20, and 22.93 respectively. The variances are 129.10, 116.56, and 75.12.

To test the normality, I used Shapiro-Wilk and d’Agostino-Pearson test together with Box plot. I have 1 outlier in G2 and G3 according to QQ plot (although Outliers in ANOVA, did not find any outliers with Alpha=0.02) and both normality tests were negative for G3. The Levene’s Tests (3 types) show that the 3 groups show that p value>0.5 (actually all of them are p>0.5) which I concluded that the samples are homogeneous (I’m not sure how it would work with the outliers). Anyhow, because the assumptions for ANOVA were not met completely, I did 2 things and compared the results:

---

1- I ran the same tests on LOG10 of my data and the results are as follows: Means for G1-3: 1.50, 1.47, and 1.34 and variance is 0.020, 0.021, 0.018 respectively. This time, only d’Agostino-Pearson test for G3 was p0.8 (homogeneous data) and the Box and Whisker diagram showed an outlier in Log G3. So my first question is if I running one way ANOVA would be recommended? Anyhow, I did the run the one way ANOVA (assuming that the variances are close enough) and p-value> 0.05 for 3 groups. I ran post hoc Contrast test and by setting c=1 for Log G1 and c=-1 for Log G2 (combination of Log G1- Log G3) , the result is ‘slightly’ significant (p=0.0498) but other compbinations show no significance. This didn’t repeat when the p-value for Tukey HSD tests were >0.05 for all 3 combinations of Log (G1-G2, G1-G3 and G2-G3). I ran non-parametric Schaich-Hamerle and Kruskal-Wallis tests followed by Contrast KW and the results of all of them are insignificant. Logically we should assume that the groups are equal and we don’t need to run the paired tests but because the mean of G3 is lower than 2 others and that the Contrast test for ANOVA was insignificant for G1-G3, I ran two paired samples t-tests and Wilcoxon Signed-Rank Test for Paired Samples between each 2 groups and surprisingly the results are as follows: T-test: Log G1-Log G2: Insignificant Log G1-Log g3: Significant Log G2-Log G3: Significant. Exactly the same results were achieved with Wilcoxon Signed-Rank Tests for Paired Samples. How can I explain this?

---

2- I tested the raw data (no Log). As a reminder, the means for Groups (G1-3) are 33.01, 31.20, and 22.93 and the variances are 129.10, 116.56, and 75.12 respectively. We have outliers (according to QQ plot) in G2 and G3 and negative normality results in G3. Since the assumptions for ANOVA are not met, I ran non parametric Kruskal-Wallis follows by Schaich-Hamerle Test and Contrast KW tests. Also I ran Welch’s Test and the result of all of the tests and post hoc test were insignificant. I even ran the resampling ANOVA test on 3 groups (bin size=0.5, Bins 0-6,Bootstrap, Iterations=10000) and p value>0.05, meaning that the samples are equal in their means. But out of curiosity, again I ran two paired samples t-tests and Wilcoxon Signed-Rank Test for Paired Samples between each 2 groups and while I expected not significance, the exact same results as was achieved with the Log values was attained: T-test and Wilcoxon signed-rank tests results : G1-G2: Insignificant G1-G3: Significant G2-G3: Significant.

---

Can you help me why the global tests (ANOVA and Kruskal-Wallis) contradict the post hoc paired tests and which one I should trust and report as my result? Is it possible to get significant results from parametric and non-parametric paired tests while the omnibus ANOVA was not significant? Am I dealing with false positive paired results?

P.S: Prior running the paired tests, the F tests were insignificant (P>0.05) for all the pairs and on both raw and Log10 data.

THANK YOU SO VERY MUCH IN ADVANCE,

Answer 1

You have taken a careful look at your data using some sensible methods. Here are a few comments that might help with some of your questions.

Normality and outliers: With only $n=7$ replications in each group, it is difficult to assess normality. It seems likely that the outlier in G3, triggered the significantly non-normal result there.

However, outliers are not necessarily an indication of non-normality unless they are extreme. It is typical of normal data to show occasional outliers. You did not explicitly say whether your outlier in G3 is extreme.

More precisely, the following simulation with 100,000 normal samples of size $n=7$ indicates that about 1/4 of of them show at least one boxplot outlier (according to the 1.5 IQR rule), with an average of about 0.3 outliers per sample.

set.seed(816);  m = 10^5
nr.out = replicate(10^5, length(boxplot.stats(rnorm(7))$out) )
mean(nr.out);  mean(nr.out >=1)
[1] 0.29368
[1] 0.24806

Non-significance of ANOVA test: Your sample variances average about 107, so standard errors of the sample means are about 4. Also, the differences among the three sample means are smaller than about 10, so I am not surprised that an ANOVA does not find significant differences among means.

More precisely, it seems that in your ANOVA, MS(Group) $\approx 202$ and MS(Error) $\approx 107,$ so that $F \approx 1.89,$ which is much smaller than the critical value $c = 3.55,$ for an F-test with degrees of freedom 2 and 18. If I'm right about that, then the ANOVA finds no significant differences.

a = c(33.01, 31.2, 22.93);  v = c(129.1, 116.56, 75.12)
f = 7*var(a)/mean(v);  f;  qf(.95, 2, 18)
[1] 1.890591
[1] 3.554557

Multiple comparisons: It is not good practice to test for significant differences among pairs of group means unless the ANOVA suggests some such differences exist.

Nevertheless, I admit it is tempting to take another look at the comparison of G1 with G3 (ignoring the existence of G2 and perhaps assuming normality), but then you should use a Welch t test to account for the differences in sample variances, and you should not make claims about the result unless the P-value is as low as .01 or .02. Looking at that difference more carefully might prompt a subsequent experiment.