Does this intuition behind why an activation function is used in a neural network make sense mathematically :

For this example lets consider a fully connected (NOT CONVOLUTIONAL) network that classifies numbers 0-9. A network WITHOUT an activation function will be able to do well on numbers that are similar to the sizes of the numbers in the training set but it will struggle if numbers appear to be darker or lighter because it is linear and cannot take both size and lightness/darkness into account. In a neural network completing the same task but WITH activation functions will be able to take into account both orientation and lightness/darkness because the weights will learn all possible relationships between the pixel values in the data set and the sigmoid(or other activation function) will then take number that are slightly lighter or darker and transform/smooth them so that the network could output the same probability as if it were the normal darkness/lightness. Does this intuition sound about right or is it incorrect in some way ?

Here is another example of what I am trying to describe if this makes more sense: I understand it is a non linearity but maybe let me try and explain what I am thinking in a different way. In a network that is detecting numbers 0-9 the network has learned for example all the relationships between the pixels of a 7 of standard darkness in all sorts of positions. Then during test time a lighter 7 comes along with slightly lighter pixel values. In one of the neurons in the input layer after the lighter 7’s input has been multiplied by the weights and biases for that neuron when it is transformed by the sigmoid function it will have approximately the same value the darker one after passing through sigmoid in neuron one would be 0.994 and the lighter one 0.992 so then the lighter one will get treated the same through all later layer of the network and get classified correctly. Does that make sense ?

  • 1
    $\begingroup$ I think it's simpler to just work it out honestly. You have a set of functions $f_i(x) = a_i x + b_i$ and your neural network without activation functions is something like $y = f_3(f_2(f_1(x)))$. If you just work out the math you'll just find out it has the form $y = m x + c$. It's impossible to create something non-linear by composing affine transformations. And it's very likely that the function of digits from pixel is not linear $\endgroup$ Commented Feb 15 at 14:57
  • $\begingroup$ @CedricMartens I have done that. I am just trying to visualize the flow of numbers when I add non linearity $\endgroup$
    – Stef
    Commented Feb 15 at 14:59
  • 1
    $\begingroup$ The nonlinearity is applied pointwise, so the "flow of numbers" is the same, except that some gradients are reduced to near zero by nonlinearities with very small derivatives, e.g., sigmoid, RuLU, etc. $\endgroup$
    – whpowell96
    Commented Feb 15 at 16:03


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.