14
$\begingroup$

In a recent paper, I fitted a three-way fixed effects models. Since one of the factors wasn't significant (p > 0.1), I removed it and refitted the model with two fixed effects and an interaction.

I've just had referees comments back, to quote:

That time was not a significant factor in the 3-way ANOVA is not of itself a sufficient criterion for pooling the time factor: the standard text on this issue, Underwood 1997, argues that the p-value for a non-significant effect must be greater than 0.25 before treatment levels of a factor can be pooled. The authors should give the relevant p-value here, and justify their pooling with reference to Underwood 1997.

My questions are:

  1. I've never heard of the 0.25 rule. Has anyone else? I can understand not removing the factor if the p-value was close to the cut-off, but to have a "rule" seems a bit extreme.
  2. This referee states that Underwood 1997 is the standard text. Is it really? I've never heard of it. What would be the standard text (does such a thing exist)? Unfortunately, I don't have access to this Underwood, 1997.
  3. Any advice when responding to the referees.

Background: this paper was submitted to a non-statistical journal. When fitting the three-way model I checked for interaction effects.

Improve this question
$\endgroup$
  • $\begingroup$ Never heard of Underwood's textbook, but this article seems to discuss the pros and cons of pooling: Pragmatics of pooling in ANOVA tables (Hines, Am. Stat. 1996). Now, I seem to remember that Sokal & Rohlf (1995) also recommend to consider very conservative values ($p\geq .25$); I need to check before posting an answer, unless better references come up. $\endgroup$ – chl Dec 6 '10 at 14:35
  • 2
    $\begingroup$ Just a comment. A guideline based on $p \geq something$ smells like a misuse of a $p$-value, in that a non-significant $p$ value is not a measure of non-evidence. Since $p$-values are uniformly distributed under the null hypothesis, why not just flip a (biased) coin? The end result is the same, and at least it's honest about being dopey. (OK, dopey is a bit strong, but you get the idea.) $\endgroup$ – user1108 Dec 6 '10 at 15:18
  • 5
    $\begingroup$ That would be an interesting response to a referee: "We thank the referee for their comments, but think they're a bit dopey" ;) Good comment though. $\endgroup$ – csgillespie Dec 6 '10 at 15:28
16
$\begingroup$

I'm guessing the Underwood in question is Experiments in Ecology (Cambridge Press 1991). Its a more-or-less standard reference in the ecological sciences, perhaps third behind Zar and Sohkol and Rohlf (and in my opinion the most 'readable' of the three)

If you can find a copy, the relevant section your referee is citing is in 9.7 on p.273. There Underwood suggests a recommended pooling procedure (so not a 'rule' per se) for non-significant factors. Its a 2-step procedure that frankly I don't quite understand, but the upshot is the p = 0.25 is suggested to reduce the probability of Type I error when pooling the non-significant factor (so nothing to do with 'time' in your example, it could be any non-sig factor).

The procedure doesn't actually appear to be Underwood's, he himself cites Winer et al 1991 (Statistical Procedures in Experimental Design McGraw-Hill). You might try there if you can't find a copy of Underwood.

Improve this answer
$\endgroup$
  • 4
    $\begingroup$ +1 Nice answer: clear, to the point, insightful, and authoritative. $\endgroup$ – whuber♦ Dec 6 '10 at 15:33
  • $\begingroup$ @Chris, do you mean "reduce the probability of Type II error" (not type I) above? The motivation for not removing factors from the model is to prevent low-powered studies permitting removal of genuine causes (i.e., the Type II of concluding the variable has no effect), while also inflating the apparent effect of parameters left in the model if they are correlated with the now-removed variable. As the side-effect will generate Type-I errors, perhaps Underwood is suggesting leaving effects in to control both Type 1 and Type II errors, i.e., maximize model validity? $\endgroup$ – tim Aug 11 '15 at 8:27
10
$\begingroup$

I loathe these sort of cut-off-based rules. I think it depends on design and what your a priori hypotheses and expectations were. If you expecting the outcome to vary with time then I'd say you should keep time in, as you would for any other 'blocking' factor. On the other hand, if you were replicating the same experiments at different times and had no reason to think the outcome would vary with time but wished to check this was the case, then having done so and found little or no evidence for it varying with time, i'd say it's quite entirely reasonable to then drop time.

I've never heard of Underwood before. It may be a standard text for 'Experiments in Ecology' (the book's title), but there's no obvious reason that experiments in ecology should be treated any differently from any other experiments in this respect, so to view it as "the standard text on this issue" seems unjustified.

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ Before the experiment, it was believed that the factor would be significant. However, it was swamped by the other two effects. I removed the factor because keeping it in didn't change the conclusions and only made the explanation harder. $\endgroup$ – csgillespie Dec 6 '10 at 15:03
  • 2
    $\begingroup$ Hmm, in that case i think i'd keep it in. I can't see why it makes the explanation harder, and as you've discovered it may be harder to explain why you dropped it than why you kept it in! $\endgroup$ – onestop Dec 6 '10 at 17:12
  • $\begingroup$ I take your point, although I don't 100% agree with it. I could easily see another referee suggesting that you should remove the factor (that's what the bio-statisticians recommend that I've spoken too). As you mentioned, when it's a grey area an arbitrary rule is not the way to go. If we wanted to mislead we would never mention that the other factor was ever involved! Completely unethical, but I suspect it happens. $\endgroup$ – csgillespie Dec 7 '10 at 10:49
1
$\begingroup$

please read the text of Underwood and references therein, it is not a rule, please read. In fact this approach is to control type II error when removing (or pooling) a "non-significanct" term in the model. What if the term you remove has a signficance level of 0.06? Are you really sure that the expected MS do not include an added effect due to the factor?. If you remove that term, you are assuming that the expected MS does not include the added effect due to that treatment BUT YOU MUST BE somewhat protected against type II error!. please excuse my poor and rush english!.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.