What's the value of ANOVA/LME?

Question

I have some data I want to analyze, for which I was curious whether one or more conditions lead to measurements significantly different from the rest. I started out by making a table of t-test p-values comparing the condition which looked different in my graphic plot with all the others.

I was told by people in my lab that that was wrong (because it constitutes data mining) and instead I should use ANOVA. So I started reading about how to use that.

Then I came here and I was told that ANOVA is generally (and especially in my case where I am doing repeated measures) inferior to linear models and that I should use linear models. Sounded good.

When finally writing up my results I used both ANOVA and LME, and got results which I found conflicting (ANOVA had a p value of 0.09, whereas LME found 3 significantly different categories) - and I asked about that in my other thread here. There I was told the following about ANOVA:

Him: "It is a well-known fact that a non-significant omnibus test does NOT imply that none of the comparisons subsumed under that omnibus test are significant."

Me: Wouldn't what you are saying amount to "it is normal for omnibus tests (here ANOVA) to lead to false negatives"?

Him: Yes.

Plus

ANOVA is, as is stated elsewhere there, sensitive to the average effect and thus can generate significant F's due to multiple differences averaging to significant with no significant paired test.

And the following about LME:

Me: so if I run lme I would have to define a priori which conditions interest me, and simply picking the most significant ones a posteriori means I'm data mining?

Him: Yes.

So my questions are:

• If both false negatives and false positives are "normal" for ANOVA - what use are ANOVA results in my case (or in any case)?

• If determining significantly different conditions (a posteriori) from LME output constitutes data mining (as would my initial table of t-test p-values) - what use are LME results in my case?

Answer 1

Every statistical method (and every other method of making decisions as well) can give false negatives and false positives. The only way to avoid mistakes is to know in advance what is correct, but, if we already knew what was correct, we wouldn't need to do any analysis!

One advantages of statistical methods is that, if you follow the rules and meet the assumptions, you can estimate the likelihood of false results. However, if you do not follow the rules or if you violate assumptions, those estimates can be wrong.

Another, perhaps more important advantage is that you can get good estimates of effect size, which (very often) ought to be of more interest than precise p values.

Data mining, in its place, is not necessarily bad. Nor is the related idea of exploratory data analysis. But you need to be clear as to whether you are doing hypothesis testing or exploration.