I know how to write up a neural network in python and I understand the "math" behind it. i,e. the value of hidden node is the linear combination of all the prior nodes and weights. This is then squashed with some sort of function (sigmoid, Relu, etc). What i'm trying to understand is, why this works out the way it is. For example, given a dataset, the intuition is that hidden layer 1 is used to detect a squiggle, hidden layer 2 detects a circle, etc etc....but I'm unable to see how this logic makes sense. How does doing a linear combination of the prior nodes and weights conclude to detecting squiggles, circles, eyes etc?

I'm trying to get a very layman terms of what goes on in the hidden layer, but at the same time, I want it to make sense with the algebra that is being used.


Your question is basically how can you detect complicated patterns using NN building blocks: linear transformations and non-linearities.

Since you specifically talk about image data, have you looked at Edge detectors and in particular Sobel operators?

Sobel operators can detect edges and they are linear. When training a CNN, you learn the weights of these edge detectors (learn the coefficients of the filter kernels). So if the first hidden layer detects edges, you can then imagine how linear combinations of these detectors can detect more complicated objects. A circle should activate the edge detectors of all directions, but only by a small amount. I guess an eye is shaped like an ellipse with a smaller circle inside.

Have you seen Chris Olah's vizs? There are more on distill.pub.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.