2
$\begingroup$

I fit a model by using fit<-glm(...). Then I perform test to see if I can exclude any variables: drop1(fit, test="Chisq))

I base my conclusion only on the p values that is returned in the drop1 function. If some variable have large PR(>Chi) values, I will let them out of the model.

I do something similar when I am testing if there are interaction. I use add1(fit.~.^2, test="Chisq"). If there any small values for PR(>Chi) I include them in my model.

However, I do those things as a routine. What is the theoretical explanation? how does the tests look mathematically?

$\endgroup$
1
  • $\begingroup$ Without knowing your scientific question it is hard to see whether this is a productive strategy. $\endgroup$
    – mdewey
    Feb 19 '17 at 22:02
1
$\begingroup$

It sounds like you are performing backwards stepwise regression.

How do the tests look mathematically? They look really bad, as you are not controlling for your familywise error rate. There is no theoretical explanation that supports your modeling strategy. The p-values that you have computed are all computed under the implicit assumption that you are not going through the data in the way that you have described. In the academic literature, there is complete agreement that such a strategy is unscientific, and the only point of contention relates to what you call it (e.g., p-hacking, data dredging, fishing).

An enjoyable paper on the topic is Gelman and Loken's The Garden of Forking Paths. More generally, Gelman's entertaining blog deals with this very regularly.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.