1
$\begingroup$

Not sure, if this is a silly question but...suppose I'm building a NN with n features

$$ x_{1}....x_{n} $$

Furthermore, suppose I strongly believe the difference of the first 2 features has meaningful impact on my output based on real world experience.

My question is: Is there any downside to manually adding an additional feature column $$x_{n+1}=x_{2}-x_{1}$$

Follow up question is: is the introduction of this new feature necessary or will the original NN be "clever enough" to discover this relationship on its own.

$\endgroup$
4
  • $\begingroup$ Adding redundant info to NN would do no harm if you dataset is not terribly small. And yes, NN should be able to discover the relationship. $\endgroup$
    – doubllle
    Commented Jun 20, 2020 at 8:30
  • $\begingroup$ @doubllle thank you. Is there a good rule of thumb on what "terribly small" is? for example, is 1000 observations considered small for 10 features? $\endgroup$
    – Kevin
    Commented Jun 20, 2020 at 23:42
  • $\begingroup$ You may find this thread interesting as well: stats.stackexchange.com/questions/350220/… $\endgroup$
    – Tim
    Commented Jun 22, 2020 at 8:33
  • $\begingroup$ thank you both. very helpful $\endgroup$
    – Kevin
    Commented Jun 23, 2020 at 0:51

1 Answer 1

1
$\begingroup$

Is there any downside to manually adding an additional feature column

No, when you have 1000 samples and only 10 features. When you provide $x_2 - x_1$ as an extra feature, you are actually doing feature engineering. Feature engineering bakes your domain knowledge into data preprocessing such that the learning can be easier for the learning model. Suppose you are using a two-layer fully connected NN with 50 hidden nodes and 1 output node. For 10 input features, you'll have 10*50 + 50 = 550 parameters to be fitted. Adding one extra feature brings 50 more parameters. Would the 50 more parameters be too many? This relates to your question in the comment.

Is there a good rule of thumb on what "terribly small" is?

This depends on your data quality, in another words, the noisier your data is, the more you'll need.
Multilayer NNs are universal approximators, we can think of that in terms of polynomial fitting. Maybe not a very good analogy, say, you use a polynomial $f(x)=w_0+w_1x+w_2x^2+w_3x^3$ as the model. If the data points you collected are truly lying on the polynomial and totally noise free, you'll only need 4 samples to uniquely determine 4 coefficients. But if the data points are noisy and the true relation is just a linear function $f(x)=w_0+w_1x$, then you are fitting an overly complex function to the data and in order to discover the linear relation, you'll need to collect more data points to overcome the overfitting.

However, in practice, you usually wouldn't know how noisy your data is. So you would want to use cross validation to reduce overfitting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.