I am trying to analyze the Berkeley Guidance Study to practice multiple regression models, which has 10 continuous variables, 1 categorical variable (with two categories) and the response variable. After trying to fit several models (using Minitab), I have some general questions and my current thoughts:

  1. Since there is one categorical variable, I'm creating interaction terms between the categorical variable and each of the continuous ones. Does it make sense in a case like this to use something like best subsets regression? I tried running it over the whole set of variables + interaction terms and found that it's hardly useful since hierarchy in the selected variables is broken (a single variable is removed and its interaction terms are left in the model).

  2. In the same line of the previous, if I had a dataset with more than one categorical variable (each with more than one category), would running best subsets only on the original variables without interaction terms be a sensible thing to do? I'm guessing it might give an idea of which predictors are useful and then try to do a stepwise regression including the interaction terms to discover which of these interactions are useful.

  3. If I find collinearity between a pair of variables, would it be sensible to remove the variable which contributes less to the model? I'd think that there is some sort of redundancy while trying to explain the dependent variable, and to have a good model I should be looking for the minimal correlation between each of the independent variables.

  4. If I'm certain that all the observations were measured correctly for any dataset, does it make sense to remove outliers just to "make the regression better?" I know that I could use Cook's distance to identify them but, is there any criterion I can use to see if I should keep the data points? I'd guess that since I'm trying to build a model that explains most of the data, it would be ok to remove the outliers as long as I mention that along with my model.

Are these ideas correct? Any insights are welcome!

  • 1
    $\begingroup$ Have you considered to use the Group Lasso? I am not familiar with minitab but you can find it for R $\endgroup$
    – Donbeo
    Oct 19, 2014 at 14:08
  • 2
    $\begingroup$ Think these questions are mostly already covered: #1; #3; #4. Note that #1 to #3 involve deciding which predictors to include in the model based on the relation of those same predictors to the response - see here for the problems with all such approaches (& why @Donbeo suggests group lasso). #2 presupposes picking best subsets is sensible per se. $\endgroup$ Oct 19, 2014 at 21:03


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.