I am trying to analyze the Berkeley Guidance Study to practice multiple regression models, which has 10 continuous variables, 1 categorical variable (with two categories) and the response variable. After trying to fit several models (using Minitab), I have some general questions and my current thoughts:
Since there is one categorical variable, I'm creating interaction terms between the categorical variable and each of the continuous ones. Does it make sense in a case like this to use something like best subsets regression? I tried running it over the whole set of variables + interaction terms and found that it's hardly useful since hierarchy in the selected variables is broken (a single variable is removed and its interaction terms are left in the model).
In the same line of the previous, if I had a dataset with more than one categorical variable (each with more than one category), would running best subsets only on the original variables without interaction terms be a sensible thing to do? I'm guessing it might give an idea of which predictors are useful and then try to do a stepwise regression including the interaction terms to discover which of these interactions are useful.
If I find collinearity between a pair of variables, would it be sensible to remove the variable which contributes less to the model? I'd think that there is some sort of redundancy while trying to explain the dependent variable, and to have a good model I should be looking for the minimal correlation between each of the independent variables.
If I'm certain that all the observations were measured correctly for any dataset, does it make sense to remove outliers just to "make the regression better?" I know that I could use Cook's distance to identify them but, is there any criterion I can use to see if I should keep the data points? I'd guess that since I'm trying to build a model that explains most of the data, it would be ok to remove the outliers as long as I mention that along with my model.
Are these ideas correct? Any insights are welcome!