ANOVA not significant but t-test significant

Question

I have survey responses from people from 10 countries (sample range from 10 to 71 response per country) and I am trying to see if there are significant differences between the people of each country and the total sample size.

I conducted 2 tests via Excel Analysis Pack (I don't have a more advanced program on my office computer and I don't have admin privilege, and I couldn't get my data to load in R):

1. ANOVA: Single factor I conducted an ANOVA across all 10 countries, and my p value was 0.225, with F < F Critical. I took the result to mean that there were no significant differences between the groups, although I would have thought there is because the means of some of the group varies quite a lot.

2. T-test (two sample assuming unequal variances) I then conducted t-tests between the sample of one country vs all the responses in all the countries. I conducted this 10 times, one for each country. After that, I realized that 3 of my countries showed significant differences (t-stat > t-critical for two tail) from the total sample size.

May I know why my ANOVA turned up negative significant difference although the individual t-tests showed that there were some differences? Am I doing my analysis wrong?

Answer 1

It is most probably due to the fact that ANOVA omnibus tests a hypothesis that there is at least one single difference between at least two specific groups, although it does not tell you which two of them differ.

At the same time your $t$-tests do something else, since they test hypotheses that a specific group's mean is different than the weighted mean of the all other groups' means.

So let's imagine that your groups consist of 50 respondents each, so in total you have 500 respondents. Then when testing one group against all other you get a $t$-test with $df = 498$, but when you test two specific groups against each other you get a $t$-test with $df = 98$. So as you see in this case you have much smaller ability to detect a significant difference. Thus, even though you have a group that has smaller or higher mean that the weighted mean of the other groups' means what implies that there is at least one group that has accordingly greater or smaller mean than it, you may not detect this difference when testing these two groups against each other due to smaller sample size that you use for testing and as result smaller power of your test. And I think this is exactly what happened in your case.