Linear independence of features in Neural Network Not sure, if this is a silly question but...suppose I'm building a NN with n features
$$
x_{1}....x_{n}
$$
Furthermore, suppose I strongly believe the difference of the first 2 features has meaningful impact on my output based on real world experience.
My question is: Is there any downside to manually adding an additional feature column $$x_{n+1}=x_{2}-x_{1}$$
Follow up question is: is the introduction of this new feature necessary or will the original NN be "clever enough" to discover this relationship on its own.
 A: 
Is there any downside to manually adding an additional feature column

No, when you have 1000 samples and only 10 features. When you provide $x_2 - x_1$ as an extra feature, you are actually doing feature engineering. Feature engineering bakes your domain knowledge into data preprocessing such that the learning can be easier for the learning model.
Suppose you are using a two-layer fully connected NN with 50 hidden nodes and 1 output node. For 10 input features, you'll have 10*50 + 50 = 550 parameters to be fitted. Adding one extra feature brings 50 more parameters. Would the 50 more parameters be too many? This relates to your question in the comment.

Is there a good rule of thumb on what "terribly small" is?

This depends on your data quality, in another words, the noisier your data is, the more you'll need.
Multilayer NNs are universal approximators, we can think of that in terms of polynomial fitting. Maybe not a very good analogy, say, you use a polynomial $f(x)=w_0+w_1x+w_2x^2+w_3x^3$ as the model. If the data points you collected are truly lying on the polynomial and totally noise free, you'll only need 4 samples to uniquely determine 4 coefficients. But if the data points are noisy and the true relation is just a linear function $f(x)=w_0+w_1x$, then you are fitting an overly complex function to the data and in order to discover the linear relation, you'll need to collect more data points to overcome the overfitting.
However, in practice, you usually wouldn't know how noisy your data is. So you would want to use cross validation to reduce overfitting.
