1
$\begingroup$

I have some data I want to analyze, for which I was curious whether one or more conditions lead to measurements significantly different from the rest. I started out by making a table of t-test p-values comparing the condition which looked different in my graphic plot with all the others.

I was told by people in my lab that that was wrong (because it constitutes data mining) and instead I should use ANOVA. So I started reading about how to use that.

Then I came here and I was told that ANOVA is generally (and especially in my case where I am doing repeated measures) inferior to linear models and that I should use linear models. Sounded good.

When finally writing up my results I used both ANOVA and LME, and got results which I found conflicting (ANOVA had a p value of 0.09, whereas LME found 3 significantly different categories) - and I asked about that in my other thread here. There I was told the following about ANOVA:

Him: "It is a well-known fact that a non-significant omnibus test does NOT imply that none of the comparisons subsumed under that omnibus test are significant."

Me: Wouldn't what you are saying amount to "it is normal for omnibus tests (here ANOVA) to lead to false negatives"?

Him: Yes.

Plus

ANOVA is, as is stated elsewhere there, sensitive to the average effect and thus can generate significant F's due to multiple differences averaging to significant with no significant paired test.

And the following about LME:

Me: so if I run lme I would have to define a priori which conditions interest me, and simply picking the most significant ones a posteriori means I'm data mining?

Him: Yes.

So my questions are:

  • If both false negatives and false positives are "normal" for ANOVA - what use are ANOVA results in my case (or in any case)?

  • If determining significantly different conditions (a posteriori) from LME output constitutes data mining (as would my initial table of t-test p-values) - what use are LME results in my case?

$\endgroup$
  • 1
    $\begingroup$ ANOVA is a linear model. For repeated measures, people may have been suggesting a multi-level model. $\endgroup$ – Peter Flom - Reinstate Monica Nov 18 '13 at 14:05
  • $\begingroup$ Then why do for instance R's stats::aov and nlme::lme give completely differently formatted results? $\endgroup$ – TheChymera Nov 18 '13 at 14:20
  • $\begingroup$ Because the two methods evolved in parallel and unknown to each other. The output looks different but means the same thing. ANOVA largely evolved in agriculture and multiple regression in geography. $\endgroup$ – Peter Flom - Reinstate Monica Nov 18 '13 at 14:23
  • $\begingroup$ Then why do people recommend linear models over anova? $\endgroup$ – TheChymera Nov 18 '13 at 14:39
  • $\begingroup$ That thread is full of comments by me; I did not recommend a linear model over ANOVA, I recommended a multi-level model over repeated measures ANOVA $\endgroup$ – Peter Flom - Reinstate Monica Nov 18 '13 at 14:48
1
$\begingroup$

Every statistical method (and every other method of making decisions as well) can give false negatives and false positives. The only way to avoid mistakes is to know in advance what is correct, but, if we already knew what was correct, we wouldn't need to do any analysis!

One advantages of statistical methods is that, if you follow the rules and meet the assumptions, you can estimate the likelihood of false results. However, if you do not follow the rules or if you violate assumptions, those estimates can be wrong.

Another, perhaps more important advantage is that you can get good estimates of effect size, which (very often) ought to be of more interest than precise p values.

Data mining, in its place, is not necessarily bad. Nor is the related idea of exploratory data analysis. But you need to be clear as to whether you are doing hypothesis testing or exploration.

$\endgroup$
  • $\begingroup$ Indeed, I am not testing a hypothesis, I am looking at a figure and trying to determine with what probability something which looks like an effect actually is one. How can I best do this? $\endgroup$ – TheChymera Nov 18 '13 at 14:22
  • 1
    $\begingroup$ The effect is one with probability 1, in your sample. I think what you are probably after is generalizing your sample to a population. Permutation/combination tests may be one way. $\endgroup$ – Peter Flom - Reinstate Monica Nov 18 '13 at 14:25
  • $\begingroup$ Yes, as I have mentioned I am trying to make a statement about my conditions. $\endgroup$ – TheChymera Nov 18 '13 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.