I'm looking for a 'not too technical' explanation on how Neural Networks with a regression output differ from running a traditional regressions (eg. OLS) on the data.
Say we want to estimate how a set of independent variables $(x_1, ..., x_n)$ are related to a dependent variable $y$. OLS estimates slope values $b_1,...,b_n$ that minimizes the sum of square errors, thus finding the 'best fit' for a given function, eg:
$$y = b_1x_1 + ... + b_nx_n + \varepsilon$$
If we were to run this data through a neural net with $n$ input nodes and an activation function that outputs a floating point value; how/why do we get a different/'better' estimate than the OLS which simply minimizes the sum squared error of the above function.
Am I correct in thinking that the answer has something to do with the neural net exploring different non-linear combinations of the x-values?