# Is it possible to get non-significant post hoc test in a significant ANOVA test while the unpair t-tests are significant? [duplicate]

I conducted One-Way ANOVA to compare means in 4 groups. The ANOVA was significant, but the post hoc tests were non-significant. Then, I applied unpair t-tests to compare each group to the lowest group. The unpair t-tests were significant. So, is it reasonable to meet something like this, and if yes, why? I would appreciate any try of help as I am a beginner in statistic and confused.

• Your entire post was bolded. This doesn't fit with the SE style; I have edited it to unbolded text. If some particular symbols, words or phrases require emphasis, bolding may be appropriate. Jul 14, 2019 at 0:50
• Why the emphasis on null hypothesis significance testing in view of the American Statistical Association's statement about its dangers? Jul 14, 2019 at 10:49

Two usages of 'paired'. The first point to discuss is that ad hoc tests are sometimes called 'paired comparisons'. If you have four levels of a factor, then you have $${4 \choose 2} = 6$$ possible comparisons to make among "pairs" of levels. However, if this is a completely randomized design, there is no 'pairing' of the data between one level of the factor and another.

So the ad hoc tests should not be 'paired' tests. (If this is an ANOVA involving blocks, then there would be pairing. I'm assuming there is no blocking, and that you have a completely randomized design with 4 levels of the factor and perhaps $$r = 10$$ subjects at each level, so there are 40 subjects randomly sampled and randomly assigned to the 4 levels.)

Two-sample vs. paired t tests. In a two-sample t test you have two independent ranom samples. Perhaps $$X_1, X_2, \dots, X_{50}$$ from $$\mathsf{Norm}(\mu_1 = 50, \sigma_1=4)$$ and, independently, $$Y_1, Y_2, \dots, Y_{50}$$ from $$\mathsf{Norm}(\mu_2 = 51, \sigma_2=4).$$ I have used R to simulate such samples below

set.seed(713)  # for reproducibility
x = rnorm(100, 50, 4)
y = rnorm(100, 51, 4)


Then if I want to see if I can detect the difference between $$\mu_1$$ and $$\mu_2,$$ I can use a pooled 2-sample t test. (Pooled is OK because $$\sigma_1 = \sigma_2$$ and the pooled test is used because of parameter var.eq=T.) The P-value is about 0.03, indicating a significant difference between the two sample means at the 5% level

t.test(x, y, var.eq=T)$p.val [1] 0.02999815  By contrast, a paired t test (inappropriate here), using data vectors x and y, looks at the 100 differences $$X_i - Y_i.$$ So the order in which outcomes are listed crucial. If the order is scrambled, a variety of different P-values can result. Below I show P-values (i) for the original order, (ii) when both vectors are sorted from smallest to largest (thus inducing positive correlation) and (iii) when one vector is sorted in increasing order and the other is sorted in decreasing order (inducing negative correlation). Of course, all three P-values are bogus, because the there is no honest pairing of the data. t.test(x, y, pair=T)$$p.val [1] 0.03546777 # (i) t.test(sort(x), sort(y), pair=T)$$p.val [1] 6.931414e-32 # (ii) t.test(sort(x), sort(y, decr=T), pair=T)$p.val
[1] 0.1239989                                 # (iii)

cor(x,y)
[1] -0.05106966              # (i) consistent with 0
cor(sort(x), sort(y))
[1] 0.9853232                # (ii)
cor(sort(x), sort(y, decr=T))
[1] -0.9863003               # (iii)


One-factor ANOVA for a completely randomized design. Theoretically, it is possible for the overall F-test of an ANOVA to show significance at the 5% level, and yet comparisons of each level with another with some kind of ad hoc tests shows no significant differences at the 5% level. In practice, this happens rarely and usually only in cases of borderline significance. This can happen because ad hoc tests use various criteria for declaring significance in order to avoid 'false discovery'

Below is an example of simulated normal data in a completely randomized design with four levels of the factor and $$r = 16$$ replications at each level. Variances are equal and population means run from 50 to 53. The overall F-test shows weak evidence of significant differences among the four sample means: P-value about 4%.

set.seed(1234)
r = 16
x1 = rnorm(r, 50, 4.5);  x2 = rnorm(r, 52, 4.5)
x3 = rnorm(r, 53, 4.5);  x4 = rnorm(r, 53, 4.5)
x = c(x1, x2, x3, x4);  g = as.factor(rep(1:4, each=r))
boxplot(x ~ g, col="skyblue2", pch=19)


anova(lm(x ~ g))

Analysis of Variance Table

Response: x
Df  Sum Sq Mean Sq F value  Pr(>F)
g          3  164.71  54.902  2.9092 0.04177 *
Residuals 60 1132.32  18.872
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


However, ad hoc comparisons are largely disappointing. The largest difference between sample means is between $$48.54$$ for level 1 and $$52.63$$ for level 4. If any difference between levels should be significant, this largest one should be. A pooled 2-sample t test between levels shows a P-value about 2%. But according to the Bonferroni method of avoiding 'false discovery' among such comparisons, we should insist on a P-value below about 1%.

t.test(x1,x4, var.eq=2)\$p.val
[1] 0.01990243


Other ad hoc methods show some significant differences, but distinctions between 'neighboring' levels are mainly unresolved.

Of course, the main difficulty here is that we do not have enough replications to assure good power to find differences of size 3 at the 5% level with population standard deviations $$\sigma = 4.5.$$ One 'power and sample size' analysis recommended $$r = 65$$ replications at each level.

• Thany you so much for the useful answer. Jul 14, 2019 at 8:51