Feature Engineering for Multiple Regression

Question

My goal:

I want to predict a single output value from multiple inputs, some of which are numerical and some of which are categorical. To do this, I plan on building a multiple-regression model (probably elastic net). In building the model, I think it makes sense to try to capture the individual linear effects of, and non-linear interactions between, my predictors.

Let's say I have 2 numerical inputs, a and b. I have an intuition that both a and b has some independent predictive value for my output, y. I also think that perhaps a-b or a/b (or some other algebraic expression) has some predictive value for my output, y.

My question:

Does it make sense to engineer these potential features and include them in the training dataset when building my model? That is, should I create columns for a-b and/or a/b in the training data, or is it a total waste of computational power since the model already accounts for these potential effects?

Analogous case using variables we ought to be familiar with:

If I want to predict lifespan from height and weight, does it make sense for me to engineer in the feature of BMI (essentially a ratio of height to weight with an adjustment), or is this redundant with building the model on the component effects of height and weight as well as the interaction effect of height by weight?

Note:

If I am using any terminology incorrectly, please forgive my naivety and notify me so that I may fix the question ASAP.

Answer 1

Good question because, although you have to notice a couple of things that I will mention below, it is a good point for many real-life applications.

First of all, just to be precise, when you say “In building the model, I think it makes sense to try to capture the individual effects of, and interactions between, my predictors.”, you have to notice that when you use a multivariate regression you are already trying to capture the interaction between the variables even if you use them in linear form and separately. More precisely, you capture their linear interactions when jointly predicting the dependent variable. Instead, what you are trying to do here is to help the model understand that there may be some non-linear effects (with a/b) between the predictors and the dependent variables. So you are not adding new information to the dataset, but you are transforming your predictors in a non-linear way and use them in a linear model. It is not wrong but, If you think there may be a non-linear relationship between your predictors and the dependent variable, then consider analyzing a non-linear model as well. You could try to see how it performs compared to a linear model.

For many application this makes some sense, because sometimes the info in a/b is valuable as you say. Personally, sometimes in my models (not necessarily linear regression though!), I used a, b and a/b. Notice however that a/b is likely to alter the scale of your data (think of cases where b is very small compared to a) and cases where a,b are large real numbers which are similar to each other so that a/b will be small. So normalize all your features subtracting the mean and scaling by the stdev each one, so that you will take everything on a similar scale and coefficients will be comparable. And check whether you have cases where the denominator approaches 0 because this will include extreme values in your predictors that will likely be troubling for the noise in the estimates of coefficients.

Notice also that when you use a,b,a-b you are likely to add multicollinearity to your model (not necessarily but likely, for simplicity think about the theoretical case where b is almost always 0 and a is a real number), so be careful with a-b, I wouldn’t use it.. but, if the correlation between a and b is high, then you can drop b and use a and a-b as predictors, this sounds better.. many times what I did in my models for similar situations is to use two predictors a and a/b-1 (or a and a-b) all normalized.