I've been studying the theory behind ANNs lately and I wanted to understand the 'magic' behind their capability of non-linear multi-class classification. This led me to this website which does a good job of explaining geometrically how this approximation is achieved.
Here's how I understood it (in 3D): The hidden layers can be thought of as outputting 3D step functions (or tower functions) that look like this:
The author goes to state that multiple such towers can be used to approximate arbitrary functions, for example:
This seems to make sense, however the author's construction is rather contrived to provide some intuition behind the concept.
However, how exactly can this be validated given an arbitrary ANN? Here is what I wish to know/understand:
- AFAIK the approximation is a smooth approximation but this 'intuition' seems to provide a discrete approximation, is that correct?
- The number of the towers seems to be based on the number of hidden layers - the above towers are created as a result of two hidden layers. How can I verify this (with an example in 3d) with only a single hidden layer?
- The towers are created with some weights forced to zero but I haven't seen this to be the case with some ANNs that I've played around with. Will it really be a tower function? Can it be anything with 4 to $n$ sides and almost approximating a circle? (The author says that is the case but leaves that as a self study).
I really wish to understand this approximation capability in 3D for any arbitrary 3D function that an ANN can be approximated with a single hidden layer - I want to see how this approximation looks to formulate an intuition for multiple dimensions?
Here's what I have in mind that I think could help:
- Take an arbitrary 3D function like $f(x_1,x_2) = x^2_1 + x^2_2 + 3$.
- Generate a training set of $(x_1,x_2)$ of say 1000 data points where many points are on the curve a few above and a few below. Those on the curve are marked as the "positive class" (1) and those not as the "negative class" (0)
- Feed this data to an ANN and visualize the approximation with one hidden layer (with about 2-6 neurons).
Is this construction correct? Would this work? How do I go about doing this? I'm not yet adept with back-propagation to implement this by myself and am looking for more clarity and direction in this regard - existing examples showing this would be ideal.