For a school project, I have implemented a 3 layer feedforward ANN with an RBF activation function that can be used to distinguish between different types of signals.

I have a demonstration coming up soon and I am afraid that I will not be able to handle even trivial questions about ANNs from the other student audience such as "why does ANN work".

A lot of people assumes that there exists some sort of underlying mathematical principle. For example, in SVM we have this construction of a hyperplane that separates different features. In HMM we have this markov model.

But for ANN...I am truly unsure what mathematical model we are talking about here. The selection of layers, neurons, learning rate or even activation function seems a bit arbitrary.

Through some search I have found that it is related to gradient descend algorithm where we are trying to find a set of weights that minimizes the potential energy function. But how can I describe this in a simple way that people can understand especially when I do not understand it too good myself?


2 Answers 2


Minimization is one of the motivating principles of neural networks; there are a lot of other ideas which may be simpler to understand, because they are related to human or animal biological systems. All networks (excluding some heuristic algorithms) have some mathematically justified motivation.

For Gradient Descent, I find the following perspective gives some simple intuition: Gradient Descent as a Random Hill Climbing algorithm. The simple idea is that you try randomly to chose some point (your weights in term of ANN) and try to assess the performance of your network; if your update is good, you can use that point (weights) at next iteration, if not - so you can rollback your changes and try again. Gradient Descent has the same idea, but your new point (weights) selection have a much stronger mathematical background. Derivative of a function is used to show you a function's slope, and using this information you can figure out in which direction you need to go without random 'jumps'.

Imagine this situation: You are alone in the mountains and you must find some place where you can keep warm at night. You go in some direction and try check if it feels warmer. If so, you can try any other direction to see if you feel even warmer still; but if not, you can go back to where you were before and try a different direction. Maybe you can see some similarities between this situation and a Random Hill Climbing algorithm. Now try imagining the same situation, but in this case you have some extra information about the mountain: you know that the lower you go, the warmer it becomes. But also, there is a very thick fog and you can't check around yourself to find a downward path. You cannot see the surrounding area to determine a distant lowest point, but can tell by using your feet where a nearby lower point is (the gradient). This example should give some basic intuition about the Gradient Descent algorithm. This also demonstrates some problems in Gradient Descent. How far do you need to go? How do you know this is the lowest (warmest) point?


It depends in the application, in general NN work because it has been shown that given the activation functions, as the number of hidden layers $N \rightarrow \infty$ a single layer feedforward network can arbitrarily approximate all functions $f$ (this is known as the Universal Approximation Theorem https://en.wikipedia.org/wiki/Universal_approximation_theorem), this has been proven for both feedforward MLPs [Hornik1991] and RBFNs [i've forgotten the reference]. For classification FF MLPs essentially work in the same manner as an SVM, and generate a convex hull by the functions approximated by the network.

The training method of the network such as gradient descent do not affect the underlying mathematics of how a network works, only how the weights are found and optimise. For example you can use other non-gradient based heuristic methods to train neural networks.

Edit- If you want to go deeper into why NNs work there is literature linking the success of NNs to the quantum mechanics and theories of everything (such as holographic theory), or https://www.youtube.com/watch?v=bLqJHjXihK8&feature=youtu.be


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.