# Linear independence of features in Neural Network

Not sure, if this is a silly question but...suppose I'm building a NN with n features

$$x_{1}....x_{n}$$

Furthermore, suppose I strongly believe the difference of the first 2 features has meaningful impact on my output based on real world experience.

My question is: Is there any downside to manually adding an additional feature column $$x_{n+1}=x_{2}-x_{1}$$

Follow up question is: is the introduction of this new feature necessary or will the original NN be "clever enough" to discover this relationship on its own.

• Adding redundant info to NN would do no harm if you dataset is not terribly small. And yes, NN should be able to discover the relationship. Commented Jun 20, 2020 at 8:30
• @doubllle thank you. Is there a good rule of thumb on what "terribly small" is? for example, is 1000 observations considered small for 10 features? Commented Jun 20, 2020 at 23:42
• You may find this thread interesting as well: stats.stackexchange.com/questions/350220/…
– Tim
Commented Jun 22, 2020 at 8:33
• thank you both. very helpful Commented Jun 23, 2020 at 0:51

No, when you have 1000 samples and only 10 features. When you provide $$x_2 - x_1$$ as an extra feature, you are actually doing feature engineering. Feature engineering bakes your domain knowledge into data preprocessing such that the learning can be easier for the learning model. Suppose you are using a two-layer fully connected NN with 50 hidden nodes and 1 output node. For 10 input features, you'll have 10*50 + 50 = 550 parameters to be fitted. Adding one extra feature brings 50 more parameters. Would the 50 more parameters be too many? This relates to your question in the comment.
Multilayer NNs are universal approximators, we can think of that in terms of polynomial fitting. Maybe not a very good analogy, say, you use a polynomial $$f(x)=w_0+w_1x+w_2x^2+w_3x^3$$ as the model. If the data points you collected are truly lying on the polynomial and totally noise free, you'll only need 4 samples to uniquely determine 4 coefficients. But if the data points are noisy and the true relation is just a linear function $$f(x)=w_0+w_1x$$, then you are fitting an overly complex function to the data and in order to discover the linear relation, you'll need to collect more data points to overcome the overfitting.