I have tried to find an answer to this question but have not found a satisfactory answer. I understand that neural networks(NNs) offer the potential to complex build non-linear models. What I don’t understand is what do NNs offer that traditional statistical non-linear models do not offer. In other words, why would I choose a NN model over a statistical model (e.g. non-linear regression) at least in some cases?


The ability to embed structural and algorithmic priors into the model.

The simplest example of this is convolutional neural networks applied to image data. The structural prior is that nearby regions of the image are more closely related / relevant to each other compared to far-away regions.

Graph convolutional networks extends this "locality" prior to arbitrary graph/network structures. 1D and 3D convolutional networks extends this prior to sound / 1D signal data and 3D scans respectively.

Powerful quadratic programming solvers have been developed. It is possible to literally embed such a QP solver as part of a neural network, inducing an algorithmic prior which says "find solutions which make use of QP". Value Iteration Networks forces a prior which says "make use of this well known RL algorithm to solve this RL problem".

Computer vision scientists can build 3D geometry into a neural network, enforcing the prior "we live in 3D euclidean space, and here is our camera model" into the architecture of the network.

  • 1
    $\begingroup$ (-1) This is definitely not something unique to NN, in fact it is found in just about all statistical models. Furthermore, I would actually argue this something NN are worse at, in that interpretation of the parameters for NN is notoriously difficult compared with many other classical statistical models. $\endgroup$ – Cliff AB Sep 21 '18 at 15:49
  • 1
    $\begingroup$ @CliffAB could you provide some references or examples of how one might augment a random forest (or a non-linear model of your choice) with domain knowledge of perspective camera geometry or take advantage of the spatial relationship between voxels in a 3D scan? $\endgroup$ – shimao Sep 21 '18 at 16:02
  • $\begingroup$ the field of spatial statistics uses domain knowledge of spatial relations to more efficiently build models from spatial data. $\endgroup$ – Cliff AB Sep 21 '18 at 16:15
  • $\begingroup$ But more generally, if you want to talk about a framework for simple inclusion of domain knowledge information, it's really hard to argue that this is what Bayesian statistics does. There's nothing that says you can't have a Bayesian NN, or think of a penalized NN as an approximation of a Bayesian inference, but to say that the strength of NN is the ability to include prior information is a very strange statement to me. $\endgroup$ – Cliff AB Sep 21 '18 at 16:20
  • $\begingroup$ I don't mean any prior on the parameters, which almost all models can make use of. Rather, I mean structural and algorithmic priors which can be built into the architecture of the network, but would otherwise be difficult to impose via a prior on parameters. $\endgroup$ – shimao Sep 21 '18 at 16:28

It's my understanding that at this point in time, there is no real solid mathematical reason for why NN have seen as much success as they have. Perhaps that's why you can't find any thing convincing at this time, although there are plenty of heuristic arguments.

The one proof that this brought up a lot (for good reason) is the "Universal Approximation Theorem"; that is, with enough neurons, any smooth function can be approximated arbitrarily well by a large enough neural network. This suggests that given enough parameters in our NN and enough data, we should be able to get arbitrarily close to the true function we are attempting to approximate.

However, the Universal Approximation Theorem alone doesn't explain the success of NN's, as NN are definitely not the only type of machine learning/statistical model that have this type of property! For a very simple alternative, you could take a linear model and simply expand the covariates to include non-linear terms and interaction effects. This can also approximate any function given enough expansions.

Now in the case of linear models, although the Universal Approximation Theorem is true, we can do the math right away to see this becomes much too data hungry to ever be of practical use. For example, suppose we have a model with $k$ covariates. A simple linear model with no parameter expansions requires fitting $k$ coefficients. If we want to include just the first order interaction effects, we are now up to $k^2$ coefficients. While this is a richer set of models than simple linear effects, it's still not that complicated. If we want to include third order effects, this requires $k^3$ coefficients. Note that we haven't even addressed adding non-linear parameter expansions yet. If $k$ is at all large, it becomes obvious that this is not going to work out well for approximating functions in which the covariates have complex interactions.

So to me, the real question is which kind of models can approximate complex relations from a finite set of data well. I think the paragraph above is fairly convincing that linear models with simple parameter expansions are not the way to go. It's my understanding that the argument for NN's is that (a) there's no convincing argument that they won't work and (b) empirically, they seem to be working quite well in a wide body of problems when one has lots of data and complex interaction of features.

  • $\begingroup$ I think you are missing the point that the main area of benefit has been in image processing.. specifically for convolutional nns.(they haven't been universally successful, and for this application it's hard to think how alternative models could fit the requirements of fitting inside convolution. $\endgroup$ – seanv507 Oct 1 '18 at 6:45
  • $\begingroup$ @seanv507: while NN, and most recently CNN's specifically, have had lots of success in image processing, my understanding is that the question is about why they've had success in so many other fields (language processing, reinforcement learning, etc., just to name a few). I think what you're getting at is that convolutions fit nicely into the NN framework, as they are simply a constrained rather than fully connected layer? But note that they can also be seen as a moving average, which is used quite often in many models (although perhaps with not as much freedom to optimize the weights) $\endgroup$ – Cliff AB Oct 1 '18 at 7:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.