4
$\begingroup$

I have a numerical dependent variable, and many independent variables. Most of my independent variables are dummy variables, but I have some categorical and numerical variables, too. I tried forward and backward model selection in R, but R returns my model empty! When I run separate simple regressions, there seems to be a significant relationship between my independent and dependent variables!

My question is: Am I going to have biased results if I run separate simple regressions with each variable, and then run a multiple regression with all the significant variables?

Improve this question
$\endgroup$
10
  • 3
    $\begingroup$ Yes, that will lead to bias. Why do you need to select variables? People often seem to assume that this is just required for some reason, but it's not clear that it ever is. 2nd, what do you mean that, "R returns my model empty"? Are you saying that the null model (no predictors) is selected? $\endgroup$
    – gung - Reinstate Monica
    Commented Jun 2, 2020 at 17:00
  • $\begingroup$ because some of the variables are not significant in predicting the dependent variable! yes, no predictors is selected. $\endgroup$
    – FnewatR
    Commented Jun 2, 2020 at 17:13
  • 2
    $\begingroup$ Who cares if some of the variables in the model are not significant? No harm will come to you, or your model, if it includes some non-significant variables. $\endgroup$
    – gung - Reinstate Monica
    Commented Jun 2, 2020 at 17:23
  • 2
    $\begingroup$ There is no need to remove those non-significant variables from your model. Doing so harms your model, whereas leaving them in does not. Forward & backward selection, simply put, cannot help with finding out which variables are actually significant. What variables are significant or not is what is reported in the original full model. $\endgroup$
    – gung - Reinstate Monica
    Commented Jun 2, 2020 at 18:03
  • 1
    $\begingroup$ @gung-ReinstateMonica, certainly will help others having the same issue. Do that please. Thank you. $\endgroup$
    – FnewatR
    Commented Jun 2, 2020 at 18:07

2 Answers 2

Reset to default
7
$\begingroup$

Yes, that will lead to bias.

The question is: Why do you need to select variables in the first place? People often seem to assume that this is just required for some reason, but it's not clear that it ever is. It may well be the case that there are some variables in the original (full) model that are not significant. But this is just fine. There is no problem if your model includes some non-significant variables. There is no need to remove those non-significant variables from the model. Doing so harms your model, whereas leaving them in does not. Forward and backward selection, simply put, cannot help with finding out which variables are 'actually significant'. Which variables are significant or not is what is reported in the original full model.

Improve this answer
$\endgroup$
7
  • 1
    $\begingroup$ To be fair, "selective inference" is a valid field of study and could be used to report significance of variables in a selected model, depending on how selection is performed. It will not be the case, however, that all selected variables are significant! $\endgroup$
    – steveo'america
    Commented Jun 2, 2020 at 18:17
  • $\begingroup$ That's reasonable, @steveo'america. $\endgroup$
    – gung - Reinstate Monica
    Commented Jun 2, 2020 at 18:23
  • 1
    $\begingroup$ Wouldn't we expect removing non-significant variables to improve generalization error by reducing overfitting to non-informative features? $\endgroup$
    – Ryan Volpi
    Commented Jun 2, 2020 at 20:17
  • 1
    $\begingroup$ @RyanVolpi, I wouldn't go that far. Understand the phenomenon you're modeling. Use your knowledge to pick variables a-priori that are likely to be worthwhile. At that point, if some aren't significant, whatever. $\endgroup$
    – gung - Reinstate Monica
    Commented Jun 3, 2020 at 0:53
  • 1
    $\begingroup$ Excellent answer but ..... it sometimes is. Two reasons that leap to mind are coillinearity and overfitting. But significance is not a good reason. $\endgroup$
    – Peter Flom
    Commented Jun 5, 2020 at 15:45
2
$\begingroup$

Adding to Gung's excellent answer:

Some reasons that you might need (or want) to eliminate variables

  • Collinearity (although there are other solutions to that problem)
  • Overfitting -- if your don't have enough data. There are various rules of thumb; one common one is that you need 10 observations for every independent variable.

Some specific reasons for keeping nonsignificant variables (beyond what Gung listed)

  • A small effect is interesting. Sometimes theory predicts a large effect and you find a small one. E.g. if you find a tribe of people where men and women are the same height, then sex will show as a nonsignificant variable. But very interesting!
  • It's involved in an interaction. There are very few cases where you want to include an interaction but not the main effects.
  • It mediates an effect
  • It is the main variable you are interested in
Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ thank you very much for this! $\endgroup$
    – FnewatR
    Commented Jun 6, 2020 at 16:18
  • 2
    $\begingroup$ I actually do think I'm dealing with both overfitting and collinearity here. About collinearity, I examined variance inflation factor of the model and some of the predictors had high vif, but other than that, I have only around 365 observations, while I have something about 40 predictors. My actual problem is that I need to know the contribution of each variable to the adjusted R2 value of my model, I found out that if my predictors aren't collinear I can just run individual simple regressions and see what share of the variance is explained by a specific variable. Mine are collinear though $\endgroup$
    – FnewatR
    Commented Jun 6, 2020 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.