Removing factors from a 3-way ANOVA table In a recent paper, I fitted a three-way fixed effects model. Since one of the factors wasn't significant (p > 0.1), I removed it and refitted the model with two fixed effects and an interaction.
I've just had referees comments back, to quote:

That time was not a significant factor
in the 3-way ANOVA is not of itself a
sufficient criterion for pooling the
time factor: the standard text on this
issue, Underwood 1997, argues that the
p-value for a non-significant effect
must be greater than 0.25 before
treatment levels of a factor can be
pooled. The authors should give the
relevant p-value here, and justify
their pooling with reference to
Underwood 1997.

My questions are:

*

*I've never heard of the 0.25 rule. Has anyone else? I can understand not removing the factor if the p-value was close to the cut-off, but to have a "rule" seems a bit extreme.

*This referee states that Underwood 1997 is the standard text. Is it really? I've never heard of it. What would be the standard text (does such a thing exist)? Unfortunately, I don't have access to this Underwood, 1997.

*Any advice when responding to the referees.


Background: this paper was submitted to a non-statistical journal. When fitting the three-way model I checked for interaction effects.
 A: I'm guessing the Underwood in question is Experiments in Ecology (Cambridge Press 1991). Its a more-or-less standard reference in the ecological sciences, perhaps third behind Zar and Sokal and Rohlf (and in my opinion the most 'readable' of the three).
If you can find a copy, the relevant section your referee is citing is in 9.7 on p.273. There Underwood suggests a recommended pooling procedure (so not a 'rule' per se) for non-significant factors. It's a 2-step procedure that frankly I don't quite understand, but the upshot is the p = 0.25 is suggested to reduce the probability of Type I error when pooling the non-significant factor (so nothing to do with 'time' in your example, it could be any non-sig factor).
The procedure doesn't actually appear to be Underwood's, he himself cites Winer et al 1991 (Statistical Procedures in Experimental Design McGraw-Hill). You might try there if you can't find a copy of Underwood.
A: I loathe these sort of cut-off-based rules. I think it depends on design and what your a priori hypotheses and expectations were. If you expecting the outcome to vary with time then I'd say you should keep time in, as you would for any other 'blocking' factor. On the other hand, if you were replicating the same experiments at different times and had no reason to think the outcome would vary with time but wished to check this was the case, then having done so and found little or no evidence for it varying with time, i'd say it's quite entirely reasonable to then drop time. 
I've never heard of Underwood before. It may be a standard text for 'Experiments in Ecology' (the book's title), but there's no obvious reason that experiments in ecology should be treated any differently from any other experiments in this respect, so to view it as "the standard text on this issue" seems unjustified.
A: please read the text of Underwood and references therein, it is not a rule, please read. In fact this approach is to control type II error when removing (or pooling) a "non-significanct" term in the model. What if the term you remove has a signficance level of 0.06? Are you really sure that the expected MS do not include an added effect due to the factor?. If you remove that term, you are assuming that the expected MS does not include the added effect due to that treatment BUT YOU MUST BE somewhat protected against type II error!. please excuse my poor and rush english!.
