In a recent paper, I fitted a three-way fixed effects models. Since one of the factors wasn't significant (p > 0.1), I removed it and refitted the model with two fixed effects and an interaction.
I've just had referees comments back, to quote:
That time was not a significant factor in the 3-way ANOVA is not of itself a sufficient criterion for pooling the time factor: the standard text on this issue, Underwood 1997, argues that the p-value for a non-significant effect must be greater than 0.25 before treatment levels of a factor can be pooled. The authors should give the relevant p-value here, and justify their pooling with reference to Underwood 1997.
My questions are:
- I've never heard of the 0.25 rule. Has anyone else? I can understand not removing the factor if the p-value was close to the cut-off, but to have a "rule" seems a bit extreme.
- This referee states that Underwood 1997 is the standard text. Is it really? I've never heard of it. What would be the standard text (does such a thing exist)? Unfortunately, I don't have access to this Underwood, 1997.
- Any advice when responding to the referees.
Background: this paper was submitted to a non-statistical journal. When fitting the three-way model I checked for interaction effects.