While studying about neural networks (still on basics - not Deep Learning etc.,) two questions came on my mind.

  • What is the reason for replacing the hard limiter function in the nodes of the multilayer perceptrons (MLPs) with smoother ones?

  • Does the Back propagation algorithm always find the best possible solution for the classification problem at hand?


1 Answer 1

  • Smooth functions has well-defined derivatives and suitable for back-propagation algorithm. There are also non-smooth ones, such as ReLU, widely used for this purpose; however, they're piecewise differentiable, while the hard-limiter function is not.

  • Back-propagation algorithm doesn't find the best solution. It is actually gradient descent, and it finds you a local optimum. Sometimes, this might be the global optimum, but frequently it is not due to the sheer size of search space.And, not all local minima are bad.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.