In an answer to this post a user suggests, based on Chapter 3 of the book "The Elements of Statistical Learning" by Hastie et. al, the following means of choosing which interaction effects to include in a model:
- Trying out all possible subsets of variables and pick the one that gives a regression with the smallest Bayesian information criterion (BIC) value
- Forward or backward stepwise selection
In the comments associated with that answer, both of these approaches are described as being bad.
So, if we shouldn't use method 1) or 2) above, how exactly do we decide what variables/interactions to use in the model? I have seen 'domain knowledge' suggested in a few places, but this seems like a bit of a cop out. Domain matter knowledge is not going to help in the very common situation in which we have no pre-existing knowledge of whether a particular interaction effect is present in nature and we are relying on the information in the data itself.
For the sake of example, suppose we have the predictors - age, gender, height, weight, experience, IQ - and the response variable salary. How do we decide what interaction effects to include/not include?
This example is probably the simplest possible scenario, as we understand all of these variables very well, and even still it is not clear how to decide which interactions to include or exclude. In other situations, we will be dealing with predictor variables for which we have no pre-existing intuition on whether interactions between them could affect the response variable.
So I am looking for a systematic method of choosing which interactions to include in a multiple regression model. How does an experienced statistician choose which interactions to include in the case when domain knowledge is not available or of no use?