Is Creating Feature Variation Helps The Model?

Question

When browsing the web I've came across a of times people creating new variation of features and adding them to the model. Let's say I have a set of engineered/raw features i've selected for my training, I can add now an additional set of log(features) and double them, or create combination of $\{*,/, \cos, \sin\}$ between every possible combination.

Before testing this I would like to know if there is a proper answer to that? I failed to find this online and would really like to know if simply by enhancing my features set like this can help my training?

Thanks

Answer 1

There is nothing magical in machine learning. Say that your feature is altitude and you want to predict the temperature. This feature only is not sufficient to obtain correct predictions. Adding features like cos(altitude) or log(altitude) will not help your model. On the contrary, you will add useless dimensionality and suffer from curse of dimensionality. That means that your model will have more struggle to find a good minima.

Performing some feature engineering (like adding feature such as f(altitude)) can be useful if you know that there is some reality behind this new feature. In this case, you will help a lot models which are mostly linear such as linear regression or feedforward neural network.

Therefore, I would say not to do it if it is totally random and do it if you know or think that your output has some dependencies with the new features.