4
$\begingroup$

I have a numerical dependent variable, and many independent variables. Most of my independent variables are dummy variables, but I have some categorical and numerical variables, too. I tried forward and backward model selection in R, but R returns my model empty! When I run separate simple regressions, there seems to be a significant relationship between my independent and dependent variables!

My question is: Am I going to have biased results if I run separate simple regressions with each variable, and then run a multiple regression with all the significant variables?

$\endgroup$
  • 3
    $\begingroup$ Yes, that will lead to bias. Why do you need to select variables? People often seem to assume that this is just required for some reason, but it's not clear that it ever is. 2nd, what do you mean that, "R returns my model empty"? Are you saying that the null model (no predictors) is selected? $\endgroup$ – gung - Reinstate Monica Jun 2 at 17:00
  • $\begingroup$ because some of the variables are not significant in predicting the dependent variable! yes, no predictors is selected. $\endgroup$ – FnewatR Jun 2 at 17:13
  • 2
    $\begingroup$ Who cares if some of the variables in the model are not significant? No harm will come to you, or your model, if it includes some non-significant variables. $\endgroup$ – gung - Reinstate Monica Jun 2 at 17:23
  • 2
    $\begingroup$ There is no need to remove those non-significant variables from your model. Doing so harms your model, whereas leaving them in does not. Forward & backward selection, simply put, cannot help with finding out which variables are actually significant. What variables are significant or not is what is reported in the original full model. $\endgroup$ – gung - Reinstate Monica Jun 2 at 18:03
  • 1
    $\begingroup$ @gung-ReinstateMonica, certainly will help others having the same issue. Do that please. Thank you. $\endgroup$ – FnewatR Jun 2 at 18:07
7
$\begingroup$

Yes, that will lead to bias.

The question is: Why do you need to select variables in the first place? People often seem to assume that this is just required for some reason, but it's not clear that it ever is. It may well be the case that there are some variables in the original (full) model that are not significant. But this is just fine. There is no problem if your model includes some non-significant variables. There is no need to remove those non-significant variables from the model. Doing so harms your model, whereas leaving them in does not. Forward and backward selection, simply put, cannot help with finding out which variables are 'actually significant'. Which variables are significant or not is what is reported in the original full model.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ To be fair, "selective inference" is a valid field of study and could be used to report significance of variables in a selected model, depending on how selection is performed. It will not be the case, however, that all selected variables are significant! $\endgroup$ – steveo'america Jun 2 at 18:17
  • $\begingroup$ That's reasonable, @steveo'america. $\endgroup$ – gung - Reinstate Monica Jun 2 at 18:23
  • $\begingroup$ Wouldn't we expect removing non-significant variables to improve generalization error by reducing overfitting to non-informative features? $\endgroup$ – Ryan Volpi Jun 2 at 20:17
  • $\begingroup$ @RyanVolpi, no. We expect removing non-significant variable to increase generalization error by inducing overfitting. It may help you to read my answer to: Algorithms for automatic model selection $\endgroup$ – gung - Reinstate Monica Jun 2 at 20:32
  • 1
    $\begingroup$ @RyanVolpi, I wouldn't go that far. Understand the phenomenon you're modeling. Use your knowledge to pick variables a-priori that are likely to be worthwhile. At that point, if some aren't significant, whatever. $\endgroup$ – gung - Reinstate Monica Jun 3 at 0:53
1
$\begingroup$

Adding to Gung's excellent answer:

Some reasons that you might need (or want) to eliminate variables

  • Collinearity (although there are other solutions to that problem)
  • Overfitting -- if your don't have enough data. There are various rules of thumb; one common one is that you need 10 observations for every independent variable.

Some specific reasons for keeping nonsignificant variables (beyond what Gung listed)

  • A small effect is interesting. Sometimes theory predicts a large effect and you find a small one. E.g. if you find a tribe of people where men and women are the same height, then sex will show as a nonsignificant variable. But very interesting!
  • It's involved in an interaction. There are very few cases where you want to include an interaction but not the main effects.
  • It mediates an effect
  • It is the main variable you are interested in
| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ thank you very much for this! $\endgroup$ – FnewatR Jun 6 at 16:18
  • 1
    $\begingroup$ I actually do think I'm dealing with both overfitting and collinearity here. About collinearity, I examined variance inflation factor of the model and some of the predictors had high vif, but other than that, I have only around 365 observations, while I have something about 40 predictors. My actual problem is that I need to know the contribution of each variable to the adjusted R2 value of my model, I found out that if my predictors aren't collinear I can just run individual simple regressions and see what share of the variance is explained by a specific variable. Mine are collinear though $\endgroup$ – FnewatR Jun 6 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.