When would a neural network outperform an OLS estimate? The optimization task is to find the operator $F(x_1, x_2, ...,x_n) \rightarrow y$. 
Question: Under which conditions should NN provide better results than LS (in terms of mean square fit error)? 
Specifically, I mean the two approaches in MATLAB (assuming column vectors):


*

*NN
net = fitnet(num_hidden_layers);
net = train(net, in_train', out_train');
out_test = net(in_test')'; 

*Least Squares: $y = w_0 + \sum_i{x_i*w_i}$
w = in_train \ out_train;
out_test = in_test * w;
Taking into account that NN approach has thresholds as non-linear operators, I tried to add non-Gaussian noise,  and non-linear input - out artificial dependencies. But couldn't find conditions in which NN would be better than LS.
 A: The second approach, which you're calling Least Squares, is essentially a single perceptron (minus the threshold/activation function). Perceptrons can only learn linearly-separable patterns, while a multi-layer neural network can act as a universal approximator. 
The classic example of this is XOR (eXclusive OR). The XOR function takes two inputs and returns true (i.e., 1) if and only if exactly one of them is true. This cannot be learnt by a perceptron, but you can build a simple neural network with 3 layers that can learn the xor function. There's a diagram of a suitable network on wikipedia. 
A: Speaking for your equations in the question/code, you are curve fitting and  your assumption about thresholding is wrong, as a result. Thresholding function is used in classification, in your case RBF is used as a basis function for NNs.
Your OLR doesn't have any nonlinear parameter, so it can't curvefit the nonlinear data unless you transform basis functions(Xi in your formula) to a nonlinear space. However, The NN both fits nonlinear and linear curves as long as the basis functions are nonlinear - that is the case in your code I presume. 
So to answer your question, The NN performs better than Linear Regression if the data is nonlinearly distributed.
