In a neural network, we have a bunch of inputs and corresponding weights + a bias which are represented by a multivariable equation. Now we squash this whole equation with a sigmoid function. How does taking the derivative of this sigmoid function help us to determine the optimized weights? I don't understand this conceptually.

  • $\begingroup$ Could you be more precise? Do you understand the notion of loss and why minimizing the loss is attractive? Then, do you understand that for simple differentiable functions of one variable, the minimization can be carried out by going down the slope of the derivative? Finally, do you understand that the sigmoid "squashes" the output so that our function can represent a probability between 0 and 1? $\endgroup$
    – bomzh
    Sep 23, 2020 at 8:43
  • $\begingroup$ I understand the differentiation of functions for multivariables and that finding the min or max but conceptually not understanding how that corresponds to determining the best coefficients/weights for the neural network... Yes, I understand that the sigmoid function squashes the output into either 0 or 1 value $\endgroup$ Sep 23, 2020 at 17:21
  • $\begingroup$ The "best" coefficients for the neural networks are, by definition, those that minimize the loss. In other words, the loss function is designed so that you are happy when it is minimized. When you say "best", you have to define mathematically what it means. "Best" according to what criterion? The loss function determines what you mean by best. $\endgroup$
    – bomzh
    Sep 29, 2020 at 7:00


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.