Suppose I have a multi regression model, dummy variables and interactions included. How am I suppose to filter through them all in order to choose the best model?

I KNOW this question was asked on various online forums, but so far none could bring an explanation a human being can understand. Moreover, I did several statistics and regression courses at university, and even they can't sum it up into a usable format.

Three tools are in my knowledge:

  1. Use R's summary() on the model and remove the insignificant variables in the t-test. (However, I know that there may be hidden relations between them, thus removing may not be the best solution)

  2. Use AIC's step() and include only those chosen variables.

  3. Build two models and compare them with ANOVA for significance (usually time consuming because I need to check each variable's effect separately)

This all looks ridiculous and unconnected to me. Are all these methods do the same? All the forums I read showcased only one of the tools but never compared to the other. Can anyone tell me where I get things wrong and, what are the steps needed to be taken in order to do feature selection on a linear model?

  • $\begingroup$ What is the sample size, and how many parameters do you have to estimate? $\endgroup$ – Tamas Ferenci Jun 9 '18 at 10:08
  • $\begingroup$ I have 100 samples, 9 continues variables and a factor which has 5 levels, which, I'll need to break to 5 separate dummy variables + create interactions (have no idea how to decide which) $\endgroup$ – Riddle-Master Jun 9 '18 at 10:30
  • $\begingroup$ Do you need to include interactions of the factor with all continuous variables or just with a single one? $\endgroup$ – Tamas Ferenci Jun 9 '18 at 10:32
  • $\begingroup$ I don't know, and I have no tool to figure out the best combinations apart from putting them by hand and comparing in ANOVA with a simple model - which is, of course, unpractical. $\endgroup$ – Riddle-Master Jun 9 '18 at 13:53

If you do univariate variable selection and use p-values to guide your decision then your overall significance level is not same as in individual case. At least sort of Bonferroni correction is in order.

But I recommend comparing two models:

1) Full model with all variables included 2) Restricted one

F-test can be used in this case


  • $\begingroup$ Agree. This is the most basic approach. However, in practive, I have 9 continues variables and another factor with 5 levels. I won't compare by hand the Full model with each variation or the Reduced model :/ $\endgroup$ – Riddle-Master Jun 9 '18 at 9:56
  • 1
    $\begingroup$ Forward or backward stepwise regression variable selection method could also be used. $\endgroup$ – Analyst Jun 11 '18 at 18:48

After talking to many people I understood the following:

  1. Unlike more sophisticated algorithms such as Trees or Random Forest, Linear Regression doesn't have much "auto selected" functions, thus I either need to get domain knowledge in the field I write the model for or to use step(), which so far, is the only auto-selection function I know about.

  2. T-test is being shown in the summary() and is useful when I want to see if adding an individual variable is a good idea.

  3. anova(), also called the partial F-test, is useful when I want to see if adding several variables together make the model better.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.