# Why does feature engineering work ?

Recently I have learned that one of ways for finding better solutions for ML problems is by creation of features. One can do that by for example summing two features.

For example, we possess two features "attack" and "defense" of some kind of hero. We then create additional feature called "total" which is a sum of "attack" and "defense". Now what appears to me strange is that even tough "attack" and "defense" are almost perfectly correlated with "total" we still gain useful information.

What is the math behind that? Or is me reasoning wrong?

Additionally, is that not a problem, for classificators such as kNN, that "total" will be always bigger than "attack" or "defense"? Thus, even after standarization we will have features containing values from different ranges?

• The practice of summing two features certainly doesn't represent "feature engineering" in general. – xji Jan 24 '18 at 22:40

You question title and the content seems mismatched to me. If you are using linear model, add a total feature in addition to attack and defense will make things worse.

### First I would answer why feature engineering work in general.

A picture is worth a thousand words. This figure may tell you some insights on feature engineering and why it works (picture source):

• The data in Cartesian coordinates is more complicated, and it is relatively hard to write a rule / build a model to classify two types.

• The data in Polar coordinates is much easy:, we can write a simple rule on $r$ to classify two types.

This tell us that the representation of the data matters a lot. In certain space, it is much easier to do certain tasks than other spaces.

### Here I answer the question mentioned in your example (total on attack and defense)

In fact, the feature engineering mentioned in this sum of attack and defense example, will not work well for many models such as linear model and it will cause some problems. See Multicollinearity. On the other hand, such feature engineering may work on other models, such as decision tree / random forest. See @Imran's answer for details.

So, the answer is that depending on the model you use, some feature engineering will help on some models, but not for other models.

• The sum need not be collinear with the addends. See for example my answer. – Kodiologist Dec 28 '17 at 22:07

The type of model we are using might not be very efficient at learning certain combinations of existing features.

For example, consider your example where features are a and d, and we are using a decision tree to predict a binary outcome that happens to be $0$ if $a+d<0$ and $1$ if $a+d\geq0$.

Since decision trees can only split along individual feature axes, our model will end up trying to build a staircase to fit a line, which will look something like this:

As you can see this will not generalize perfectly to new data. We can have circles above the true decision line that are under our decision boundary and vice versa for crosses.

However, if we add a+d as a feature then the problem becomes trivial for a decision tree. It can ignore the individual a and d features and solve the problem with a single a+d<0 decision stump.

However, if you were using linear regression, then your model would be perfectly capable of learning $a+d$ without adding an additional feature.

In summary, certain additional features can help depending on the type of model you are using, and you should be careful to consider both the data and the model when engineering features.

• This is exactly the point. Choice of features and choice of model must be considered together. It is a common pitfall to try and reason about feature selection without considering the type of model being used. – Imran Dec 28 '17 at 18:09
• For example if you tried the same thing with linear regression then a and d would suffice and adding a+d as a feature wouldn't make a difference. – Imran Dec 28 '17 at 18:18
• I have updated my answer to make this more explicit. – Imran Dec 28 '17 at 18:21
• Furthermore, splitting across the diagonal line requires one split. The staircase you drew "uses up" seven splits. – Acccumulation Dec 28 '17 at 19:52

A constructed feature like total can still be predictively useful if it isn't strongly correlated with other features in the same model. total in particular need not be strongly correlated with attack or defense. For example, if attack is (8, 0, 4) and defense is (1, 9, 6), then the correlation of total with attack is 0 and the correlation of total with defense is $\frac{1}{7}$.

Additionally, is that not a problem, for classificators such as kNN, that "total" will be always bigger than "attack" or "defense"? Thus, even after standarization we will have features containing values from different ranges?

If you want to standardize your predictors, you should do it after they've all been constructed.

• is this really true? Certainly, in a simple linear model, it is not: the matrix [attack, defense, total] is of course rank 2. I could imagine in something like a penalized linear model it could make a difference, but that's based on intuition rather than fully working through it. Can you explain why if attack and defense are not strongly correlated with total (which happens when attack and defense are strongly negatively correlated), why total can be helpful? – Cliff AB Dec 30 '17 at 17:41
• @CliffAB In hindsight, I was a bit glib here. I was right in saying that a constructed feature can be useful when it's not strongly correlated with other predictors, and that total need not be strongly correlated with attack or defense, but you would never use two predictors and their sum in the same model, because of the linear dependency, with implies a strong correlation between some two of the three. – Kodiologist Dec 30 '17 at 17:51