8
$\begingroup$

I'm implementing a logistic regression model in R and I have 80 variables to chose from. I need to automatize the process of variable selection of the model so I'm using the step function.

I've no problem using the function or finding the model, but when I look at the final model I find that some of the variables chosen by the step function are not significant (I look at this using the summary function and looking at the fourth column in $coef, this is the Wald Test). This is a problem because I need all the variables included in the model to be significant.

Is there any function or any way to get the best model based on AIC or BIC methods but that also consider that all the coefficients must be significant? Thanks

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ It seems to me that you have two competing goals here. Goal 1 is to have a model where all variables are significant and Goal 2 is to have the best model based on AIC/BIC. $\endgroup$
    – TrynnaDoStat
    Feb 2, 2015 at 20:43
  • $\begingroup$ @TrynnaDoStat Thanks for your answer but I don't think that's the problem. When I do a model using SPSS Modeler I use as input all the variables and the output is the best model chosen by stepwise forward method and all the coefficients are significant (at least by Wald Test) $\endgroup$
    – Dan
    Feb 2, 2015 at 20:51
  • 7
    $\begingroup$ Looking for models where all variables are significant sounds like data dredging. Statistical significance has its original interpretation when you have prespecified model form and variables. Once you do model search/selection, all the statistics can no longer be interpreted as is because they are a result of a model selection procedure. Meanwhile, if the model you got after stepwise selection has some insignificant coefficients, that need not mean it is a bad model. Perhaps the effect sizes are so big that they compensate the lack of stat. significance (this is a rough statement, I know). $\endgroup$
    – Richard Hardy
    Feb 2, 2015 at 20:52
  • $\begingroup$ I am curious - why do you think you need to select a subset of predictors? also are your predictors numeric (e.g. income) or nominal (e.g. male/female/other) $\endgroup$
    – probabilityislogic
    Dec 14, 2019 at 7:01

1 Answer 1

Reset to default
8
$\begingroup$

Using stepwise selection to find a model is a very bad thing to do. Your hypothesis tests will be invalid, and your out of sample predictive accuracy will be very poor due to overfitting. To understand these points more fully, it may help you to read my answer here: Algorithms for automatic model selection.

The stepAIC function is selecting a model based on the AIC, not whether individual coefficients are above or below some threshold as SPSS does. However, the AIC can be understood as using a specific alpha, just not .05. Instead, it's approximately .157. For more on that, see @Glen_b's answers here: Stepwise regression in R – Critical p-value.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thanks! I understand perfectly what's your point. But suppose that I need to do some automatic selection of the model, do you have any recommendation? I'm talking about the best of the bads ideas I know, but is important that the selection is automatized. $\endgroup$
    – Dan
    Feb 2, 2015 at 21:07
  • 3
    $\begingroup$ If you want valid hypothesis tests / p-values, you cannot use automatic selection. There isn't really a way around that. If you want out of sample predictive accuracy, you can use the LASSO & select for lambda by cross validation. $\endgroup$
    – gung - Reinstate Monica
    Feb 2, 2015 at 21:11
  • $\begingroup$ Thank you very much! But now I have another problem, if it is no to much to ask I will like to ask you a new question. I'm running this code (using glmnet package): lasso<-cv.glmnet(x,factor(y),family="binomial") coef(lasso,lambda=lasso$lambda.1se) But the coefficients are all 0 and the model is only an intercept. And the same happends if I use: coef(lasso,lambda=lasso$lambda.min) Is something wrong with my code or this is the correct answer for LASSO? Thanks in advance. $\endgroup$
    – Dan
    Feb 3, 2015 at 1:04
  • $\begingroup$ With a little of research i could get a model with coefficients different from 0 typing the following lines: cv=cv.glmnet(x,y) model<-glmnet(x,y,family="binomial",lambda=cv$lambda.min) coef<-predict(model, type="coefficients") I didn't use cv$lambda.1se because I only got 1 coefficient diferent from 0. ¿Is it ok? $\endgroup$
    – Dan
    Feb 3, 2015 at 1:48
  • $\begingroup$ Possibly. It's hard to say. How much data do you have? It may be that the intercept is the only thing you have enough data to estimate accurately without overfitting. It may also be that the true values of your coefficients is very close to 0. $\endgroup$
    – gung - Reinstate Monica
    Feb 3, 2015 at 2:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.