# Feature Engineering for Multiple Regression

My goal:

I want to predict a single output value from multiple inputs, some of which are numerical and some of which are categorical. To do this, I plan on building a multiple-regression model (probably elastic net). In building the model, I think it makes sense to try to capture the individual linear effects of, and non-linear interactions between, my predictors.

Let's say I have 2 numerical inputs, a and b. I have an intuition that both a and b has some independent predictive value for my output, y. I also think that perhaps a-b or a/b (or some other algebraic expression) has some predictive value for my output, y.

My question:

Does it make sense to engineer these potential features and include them in the training dataset when building my model? That is, should I create columns for a-b and/or a/b in the training data, or is it a total waste of computational power since the model already accounts for these potential effects?

Analogous case using variables we ought to be familiar with:

If I want to predict lifespan from height and weight, does it make sense for me to engineer in the feature of BMI (essentially a ratio of height to weight with an adjustment), or is this redundant with building the model on the component effects of height and weight as well as the interaction effect of height by weight?

Note:

If I am using any terminology incorrectly, please forgive my naivety and notify me so that I may fix the question ASAP.

• I like the question. Regressing on 'a' while controlling for 'b' does not cover terms such as a-b, a/b, or the term that strictly would be called the "interaction" term, a*b. So if you have a basis for including one or more of them, they could improve upon your model. (Kudos for cross-validating.) I agree with @Fr1 that you need to watch for collinearity when including such terms. You might find yellowbrickstats.com/partial.htm a help even though much of it is quite basic. Cheers~ – rolando2 Aug 16 at 1:13

Good question because, although you have to notice a couple of things that I will mention below, it is a good point for many real-life applications.

First of all, just to be precise, when you say “In building the model, I think it makes sense to try to capture the individual effects of, and interactions between, my predictors.”, you have to notice that when you use a multivariate regression you are already trying to capture the interaction between the variables even if you use them in linear form and separately. More precisely, you capture their linear interactions when jointly predicting the dependent variable. Instead, what you are trying to do here is to help the model understand that there may be some non-linear effects (with a/b) between the predictors and the dependent variables. So you are not adding new information to the dataset, but you are transforming your predictors in a non-linear way and use them in a linear model. It is not wrong but, If you think there may be a non-linear relationship between your predictors and the dependent variable, then consider analyzing a non-linear model as well. You could try to see how it performs compared to a linear model.

For many application this makes some sense, because sometimes the info in a/b is valuable as you say. Personally, sometimes in my models (not necessarily linear regression though!), I used a, b and a/b. Notice however that a/b is likely to alter the scale of your data (think of cases where b is very small compared to a) and cases where a,b are large real numbers which are similar to each other so that a/b will be small. So normalize all your features subtracting the mean and scaling by the stdev each one, so that you will take everything on a similar scale and coefficients will be comparable. And check whether you have cases where the denominator approaches 0 because this will include extreme values in your predictors that will likely be troubling for the noise in the estimates of coefficients.

Notice also that when you use a,b,a-b you are likely to add multicollinearity to your model (not necessarily but likely, for simplicity think about the theoretical case where b is almost always 0 and a is a real number), so be careful with a-b, I wouldn’t use it.. but, if the correlation between a and b is high, then you can drop b and use a and a-b as predictors, this sounds better.. many times what I did in my models for similar situations is to use two predictors a and a/b-1 (or a and a-b) all normalized.

• Hmmm... I do not think I am sold on your first point. Having done tons of two-way ANOVAs (which are a special case of linear regression), I think that interactions must be considered specifically, and are not already accounted for by main effects of factors. I remember stats teachers harping that if there is a significant interaction effect, the main effects of factors should be completely ignored since they are meaningless in isolation from one another. I agree with your final point about the problems of introducing colinearity, which is one of the reasons I want clarity on how to proceed. – Jayden.Cameron Aug 15 at 11:51
• Yes you are right, indeed, because de facto it is not likely that a linear model will capture the non-linear interactions. But I just wanted to highlight that you can’t use the statement “I want to capture the interaction between the regressors when predicting the dependent variable” because the reader says “the regression typically aims to do this”. Here you must say “I want the model to capture the NON-LINEAR relationship between the regressors when predicting the dependent variable”. It was just a way to improve your language. The meaning was 100% clear, and it was a good point – Fr1 Aug 15 at 11:57
• Thanks for your comment. I made an edit to try to make this more clear. If you don't mind, could you clarify why you would want to normalize the data to put it all on the same scale? Back to the 'real-life' example, wouldn't that make interpreting coefficients more difficult? If height is cm, weight is kg, and lifespan is years, the size and sign of coefficients is easy to understand, but if you normalize these, now the coefficients of the model are more difficult to explain in plain language. What would a coefficient of -2 for normalized weight indicate for lifespan? – Jayden.Cameron Aug 15 at 15:19
• @Jayden.Cameron no don’t worry you did not need to edit to incorporate my comment, it was just a remark for the sake of punctuality. To scale the variables means to subtract from each variable its sample mean and divide them by the sample standard dev. Nothing more, so you have the same variables, but they are normalized so that each variable has the same sample mean of 0 and the same unitary st dev.. this is just good practice when you have a variable expressed in kg for example and one other that is expressed as the ratio of kg/kg so it is just a simple scalar.. nothing more than this.. – Fr1 Aug 15 at 15:43
• To sum up, It is ok to include a/b among regressors, I just clarified and anticipated some things that you may want to look after when including a/b – Fr1 Aug 15 at 15:46