3
$\begingroup$

In feature engineering, should I drop all features that can be calculated using other features?

For example, let us say that we have this dataset:

f1    f2    f3        f4     target
1     4     0.25      5      9
3     6     0.5       9      8
5     8     0.625     13     12
2     8     0.25      10     15

In this dataset,

f3 = f1 / f2

f4 = f1 + f2

  • Should we drop f3 and f4?

  • Does the decision depend on the size of the dataset?

  • Does the decision depend on the algorithm used (Random Forest, Neural Network, etc)?

  • Does the decision depend on the machine-learning type: regression, classification, etc?

  • Does the decision depend on the formula that links the features? For example, the formula that links f3 to f1 and f2 is simple division.

Can you please also recommend resources to learn more about this topic?

$\endgroup$

1 Answer 1

6
$\begingroup$

TL;DR It depends on the algorithm. In some cases it won't make a difference, in other it will be recommended, yet in other cases it may be harmful.

If you use algorithms that are based on linear combinations of features ($\beta_1 f_1 + \beta_2 f_2 + \dots$), like generalized linear models (including linear regression, logistic regression), or densely connected neural networks, then it doesn't make sense to include variables like $f_4 = f_1 + f_2$, since this is already a build-in feature of the model. Moreover, you would get into problems with multicollinearity (by definition, such feature is linearly dependent on others). Including features such as $f_5 = f_1 f_2$ means using interaction terms and is commonly used in statistics and $f_3 = f_1 / f_2$ is just an interaction in disguise.

On another hand, with algorithms like random forest you could use exactly the same variable multiple times and it wouldn't matter, since what decision trees do is they look at one variable at a time to make a split. Adding such features when using random forest may make sense and simplify the task for the algorithm.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.