OLS Regression versus Neural Network with regression output

Question

I'm looking for a 'not too technical' explanation on how Neural Networks with a regression output differ from running a traditional regressions (eg. OLS) on the data.

Say we want to estimate how a set of independent variables $(x_1, ..., x_n)$ are related to a dependent variable $y$. OLS estimates slope values $b_1,...,b_n$ that minimizes the sum of square errors, thus finding the 'best fit' for a given function, eg:

$$y = b_1x_1 + ... + b_nx_n + \varepsilon$$

If we were to run this data through a neural net with $n$ input nodes and an activation function that outputs a floating point value; how/why do we get a different/'better' estimate than the OLS which simply minimizes the sum squared error of the above function.

Am I correct in thinking that the answer has something to do with the neural net exploring different non-linear combinations of the x-values?

Answer 1

Am I correct in thinking that the answer has something to do with the neural net exploring different non-linear combinations of the x-values?

Yes, you are exactly correct. OLS can only learn a linear combination of the input features, whereas a neural network with one or more hidden layers can learn any function of the input features, including non-linear functions. In this sense the neural network is a universal function approximator.

In practice, the neural network's ability to accurately approximate the underlying function of the data will increase with more neurons and more hidden layers. However, increasing the size of the neural network comes with the danger of overfitting the data, in which case it may be ideal to reduce its size again or to add regularization.