Suppose I have a multi regression model, dummy variables and interactions included. How am I suppose to filter through them all in order to choose the best model?
I KNOW this question was asked on various online forums, but so far none could bring an explanation a human being can understand. Moreover, I did several statistics and regression courses at university, and even they can't sum it up into a usable format.
Three tools are in my knowledge:
summary()on the model and remove the insignificant variables in the t-test. (However, I know that there may be hidden relations between them, thus removing may not be the best solution)
step()and include only those chosen variables.
Build two models and compare them with ANOVA for significance (usually time consuming because I need to check each variable's effect separately)
This all looks ridiculous and unconnected to me. Are all these methods do the same? All the forums I read showcased only one of the tools but never compared to the other. Can anyone tell me where I get things wrong and, what are the steps needed to be taken in order to do feature selection on a linear model?