Andrew Ng in one of his deep learning course videos says that the sigmoid function acts as a slow learner in a neural network.

My intuition is that the sigmoid as an activation function contributes very less to the change in the input for the tail part (after 4 in the fig). Sigmoid

Please help me understand the intuition of the statement in detail.

Thanking in anticipation.

  • $\begingroup$ Slow compared to what? $\endgroup$
    – Dave
    Sep 13 '20 at 5:02
  • $\begingroup$ Slow compared to ReLU, I assume... for the reasons given in the question itself. $\endgroup$ Sep 13 '20 at 7:43
  • 3
    $\begingroup$ This may help you: stats.stackexchange.com/questions/126238/… $\endgroup$
    – Javier TG
    Sep 13 '20 at 11:30
  • $\begingroup$ Sorry for the ambiguity, but yes, slow in comparison with other activation functions like ReLU. $\endgroup$
    – CRBelhekar
    Sep 13 '20 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.