Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
While studying about neural networks (still on basics - not Deep Learning etc.,) two questions came on my mind.
What is the reason for replacing the hard limiter function in the nodes of the multilayer perceptrons (MLPs) with smoother ones?
Does the Back propagation algorithm always find the best possible solution for the classification problem at hand?
Smooth functions has well-defined derivatives and suitable for back-propagation algorithm. There are also non-smooth ones, such as ReLU, widely used for this purpose; however, they're piecewise differentiable, while the hard-limiter function is not.
Back-propagation algorithm doesn't find the best solution. It is actually gradient descent, and it finds you a local optimum. Sometimes, this might be the global optimum, but frequently it is not due to the sheer size of search space.And, not all local minima are bad.
