Should I add a new difference(z = x1-x2) feature into model?

Question

Recently, I am thinking about this question:should I add new features based on raw features' differences?

Setting

Suppose I have 50k data and 20 features and it's a regression task. In data science practice, we usually add new features based on raw features. However, I don't know when we should add a new feature z(z = x1 - x2) into our model.

My Throughts

Here is my understanding:since feature is going to be dumped in models, so whether a feature works fine depends on both feature and model.

Let's take linear regression as an example:

head(mtcars)
fit1 = lm(mpg~ cyl+disp +hp +vs, data = mtcars)
summary(fit1)$adj.r.squared

data_add = cbind(mtcars,'c1' = mtcars$disp - mtcars$hp)
fit2 = lm(mpg~ cyl+disp + hp +vs + c1, data = data_add)
summary(fit2)$adj.r.squared
summary(fit2)

add_noise <- function(n){
  res = NULL
  for(i in 1:n){
    data_add_noise = cbind(mtcars,'c1' = mtcars$disp - mtcars$hp + rnorm(nrow(mtcars),0,1))
    fit3 = lm(mpg~ cyl+disp + hp +vs + c1, data = data_add_noise)
    res = c(res,summary(fit3)$adj.r.squared)
  }  
  return(mean(res))
}
add_noise(10000)

Outputs:

> summary(fit1)$adj.r.squared
[1] 0.7359967
> summary(fit2)$adj.r.squared
[1] 0.7359967
> add_noise(10000)
[1] 0.7359121

In linear regression, if we put z = x1-x2 into our model, we will get a singular design matrix, which means R won't use z to fit coefficients. In other words, new feature z won't give any boost to our model performance.

If we use z = x1- x2 + rnorm(n=1,mean = 0,sd = 1) into our model, we will decrease our model performance since we introduce additional noise into our model.

However, if we use lgbm/xgboost/rf models, since tree model split nodes based on infomation gain/infomation gain ratio/gini impurity, our new feature z = x1 - x2 may helps with our model performance.

Summary

Whether we should add our new difference feature(z = x1- x2) into our model depends on model we use. I will be very grateful to see any other ideas!

Answer 1

The defining characteristic of a linear model is that it is linear. This means that the outcome $y$ is modeled as a linear function of the noiseless features $x_1, x_2$.

$$ y = \beta_0 + \beta_1 x_1 + \beta_2 x_2+ \epsilon $$

Suppose we add a noiseless feature $x_3=x_1 - x_2$. If we look at how this model is expressed, it should be obvious that this is not any different than our original model. $$\begin{align} y &= \beta_0 + \tilde{\beta}_1 x_1 + \tilde{\beta}_2 x_2 + {\beta}_3 (x_1 - x_2)+ \epsilon \\ y &= \beta_0 + (\tilde{\beta}_1 + {\beta}_3) x_1 + (\tilde{\beta}_2 - {\beta}_3) x_2+ \epsilon \\ y &= \beta_0 + \beta_1 x_1 + \beta_2 x_2+ \epsilon \\ \end{align}$$ In other words, the coefficient on $x_3$ is not identified in this model because it's exactly a linear combination of $x_1$ and $x_2$.

Your example uses noise $x_3 = x_1 - x_2 + \eta$ to avoid non-identification. However, this amounts to adding a coeffiicent for the noise $\eta$: $$\begin{align} y &= \beta_0 + \tilde{\beta}_1 x_1 + \tilde{\beta}_2 x_2 + {\beta}_3 (x_1 - x_2 + \eta) + \epsilon\\ y &= \beta_0 + (\tilde{\beta}_1 + {\beta}_3) x_1 + (\tilde{\beta}_2 - {\beta}_3) x_2 + {\beta}_3\eta + \epsilon \\ y &= \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 \eta + \epsilon \\ \end{align}$$

In other words, the noise $\eta$ is a third feature provided to the model. Noise is assumed to be unrelated to $y$, so we know that the true effect of $\eta$ on $y$ is zero; including $\eta$ will likely hurt predictions whenever $\hat{\beta}_3 \neq 0$.

Conclusion: don't add $x_1-x_2+\eta$ to a linear regression model because it has no new information about $y$.

---

The tree ensemble model (random forest, xgboost) is nonlinear: for any binary split, the daughter nodes yield distinct constant functions. The effect of many such binary splits is to divide the feature space into a number of axis-aligned rectangles, each with a different estimate.

Arbitrarily many binary, axis-aligned splits can approximate a complex boundary by using simpler shapes. The classic example is to consider a binary classification task with a perfect linear decision boundary on the line $x_1 - x_2 > c$. This manifests as a diagonal split. Clearly a single axis-aligned split can't approximate a diagonal very well, but many axis aligned splits, you can make a "stair-step" shape that can approximate the diagonal arbitrarily well. Likewise, the same is true for approximating relationships like logarithms, quadratics, sinusoids, etc.

On the other hand, adding a feature $x_1 - x_2$ to the feature set could improve the model because a binary split will be able to exactly recover $x_1 - x_2 > c$. This kind of feature engineering can improve the model when you know in advance that this feature is useful. On the other hand, the whole point of using advanced models like random forest or boosted trees is to recover useful functions when we don't know exactly how all of the features are related to the outcome.

Conclusion: adding $x_1 - x_2$ can improve the model if $x_1 - x_2 > c$ is important to $y$.

More information: Consequences of adding transformed features columns for random forests and lasso?