0
$\begingroup$

i am doing stepwise for logistic regression then the p-values of all variables selected were high then 0.05. According to this publication Stepwise regression in R – Critical p-value I changed the code to the following

step(glm(y~.,data=mydat,family="binomial"),direction="both",k=9)
 9.22952=qchisq(0.05,3.84,lower.tail=FALSE)

but the problem persist mean that i have the following p-values

0.21
0.19

What shall i do? Thanks a lot in advance for any help.

$\endgroup$
  • $\begingroup$ qchisq(0.05, 1, lower.tail = FALSE) does not equal 9 in my version of R. I suppose it is pointless telling you that the post you link to does say this (stepwise modelling) is a bad idea anyway. $\endgroup$ – mdewey Dec 4 '16 at 14:23
  • $\begingroup$ yes mdewey you are right i modified the post, why stepwise modelling is a bad idea ? $\endgroup$ – prep Dec 4 '16 at 14:29
  • $\begingroup$ I still think your call of qchisq is wrong. $\endgroup$ – mdewey Dec 4 '16 at 14:30
  • $\begingroup$ mdewey this is what i found in the link above ! $\endgroup$ – prep Dec 4 '16 at 14:52
7
$\begingroup$

I think there is a more serious issue here than the use of stepwise regression

but the use is a must for me ! how can i get 0.05 p values means significant variables using stepwise ?

Science is not a quest for p < 0.05. Science is a quest for discovering repeatable and understandable patterns in our world. If you go into research looking for p < 0.05, you will find it with enough effort. Unfortunately, to do so, you sell out the soul of true science, and your results will no longer be scientific.

The idea between the p < 0.05 threshold is to guarantee that at most $5\%$ of research findings are false positives. But this guarantee makes a lot of assumptions about the honesty and integrity of the scientists using the statistical tools. Dredging your data to find p < 0.05 is about the worst thing you can do, it annihilates all of the guarantees the statistical framework is supposed to provide.

So yes, we could tell you how to torture your data until you get the magical p < 0.05, but we will not do so. To do so would be to sell out the thing we truly love, science.

$\endgroup$
  • 2
    $\begingroup$ I agree with Frank. You need a deeper understanding of the procedures you are using. You need to understand why p-values are invalidated by stepwise selection. There are many, many explanations of this in the history of this site. There is no simple fix, and there is no recipe for correct science. Only a deeper understanding will help you. $\endgroup$ – Matthew Drury Dec 4 '16 at 17:43
  • $\begingroup$ Honestly, if you are having trouble dredging for p-values meeting an arbitrary threshold, it seems likely that the scientifically honest conclusion is that your desired research result is probably false. $\endgroup$ – Matthew Drury Dec 4 '16 at 17:44
  • $\begingroup$ so i can use what in place of stepwise or what comments i should put for these p-values ? other things to add are that i am searching the influence of quantitative variables in a qualitative variable which is (normal/abnormal) this is why i am using logistic regression and stepwise selection ! $\endgroup$ – prep Dec 4 '16 at 17:44
  • 2
    $\begingroup$ Perhaps the OP would like to look at stats.stackexchange.com/questions/20836/… which is one of the 'many, many explanations' $\endgroup$ – mdewey Dec 4 '16 at 17:51
  • 1
    $\begingroup$ Just to point this out a p<0.05 threshold does certainly not guarantee that at most 5% of research findings are false positives. I guarantees that at most 5% of p-values will be <0.05 under the null hypothesis, but depending on how many of the investigated null hypotheses hold the false positive rate could be way higher than 5%. $\endgroup$ – Björn Dec 5 '16 at 11:06
5
$\begingroup$

Without penalizing for the variable selection algorithm your results are very likely to be overstated, misleading, and P-values will be too low and confidence intervals too narrow.

$\endgroup$
  • $\begingroup$ hi frank, i don't undersand what shall i do ? $\endgroup$ – prep Dec 4 '16 at 14:51
  • 4
    $\begingroup$ Avoid stepwise regression until you understand all the ramifications of data dredging. My course notes go into some detail: biostat.mc.vanderbilt.edu/rms $\endgroup$ – Frank Harrell Dec 4 '16 at 14:52
  • $\begingroup$ but the use is a must for me ! how can i get 0.05 p values means significant variables using stepwise ? thank you in advance $\endgroup$ – prep Dec 4 '16 at 14:57
  • 3
    $\begingroup$ Learning any field takes time. I would not try to be an expert in your field with a few minutes of study. $\endgroup$ – Frank Harrell Dec 4 '16 at 15:02
  • 4
    $\begingroup$ How can the use of stepwise regression be "a must"? One usually starts with a question to be investigated and then identifies an appropriate statistical method. Starting with a method (that also happens to be inappropriate for any other purpose than somehow generating p<0.05 no matter whether it means anything) is the wrong way around. $\endgroup$ – Björn Dec 5 '16 at 11:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.