Why use gradient descent with neural networks? 
*

*When training a neural network using the back-propagation algorithm, the gradient descent method is used to determine the weight updates. My question is: Rather than using gradient descent method to slowly locate the minimum point with respect to a certain weight, why don't we just set the derivative $\frac{d(\text{Error})}{dw}=0$, and find the value of weight $w$ which minimizes the error?

*Also, why are we sure that the error function in back-propagation will be a minimum? Can't it turn out the error function is a maximum instead? Is there a specific property of the squashing functions that guarantees that a network with any number of hidden nodes with arbitrary weights and input vectors will always give an error function that has some minima?
 A: In Newton-type methods, at each step one solves $\frac{d(\text{error})}{dw}=0$ for a linearized or approximate version of the problem. Then the problem is linearized about the new point, and the process repeats until convergence. Some people have done it for neural nets, but it has the following drawbacks,


*

*One needs to deal with second derivatives (the Hessian, specifically Hessian-vector products).

*The "solve step" is very computationally expensive: in the time it takes to do a solve one could have done many gradient descent iterations.


If one uses a Krylov method for the Hessian solve, and one does not use a good preconditioner for the Hessian, then the costs roughly balance out - Newton iterations take much longer but make more progress, in such a way that the total time is roughly the same or slower than gradient descent. On the other hand, if one has a good Hessian preconditioner then Newton's method wins big-time. 
That said, trust-region Newton-Krylov methods are the gold-standard in modern large-scale optimization, and I would only expect their use to increase in neural nets in the upcoming years as people want to solve larger and larger problems. (and also as more people in numerical optimization get interested in machine learning)
A: *

*Because we can't. The optimization surface $S(\mathbf{w})$ as a function of the weights $\mathbf{w}$ is nonlinear and no closed form solution exists for $\frac{d S(\mathbf{w})}{d\mathbf{w}}=0$.

*Gradient descent, by definition, descends. If you reach a stationary point after descending, it has to be a (local) minimum or a saddle point, but never a local maximum.
A: Regarding Marc Claesen's answer, I believe that gradient descent could stop at a local maximum in situations where you initialize to a local maximum or you just happen to end up there due to bad luck or a mistuned rate parameter.  The local maximum would have zero gradient and the algorithm would think it had converged.  This is why I often run multiple iterations from different starting points and keep track of the values along the way.
