Is backpropagation required for optimizing MLPs? [duplicate]

Question

In a neural network (MLP), we can define the error function during training the difference between the target output and the output for the current input. Now we use backpropagation like so:

1. Calculate the error of the current layer (starting with the output layer going back).
2. Calculate the gradient of the current layer which is the derivative of the total error (from the output layer) derived by the weights of the current layer.
3. Update the weights of the current layer by the gradient multiplied with a learning rate (here we use simple gradient descend).
4. Calculate the gradient of the previous layer which is the derivative of the total error (from the output layer) derived by the weights of the current layer, where we fix the weights of the next layers (which are closer to the output).
5. Update the weights of the current layer by the gradient multiplied with a learning rate (here we use simple gradient descend). ... Continue with 4.

Assuming each layer as N neurons, each layer has N^2 weights. Let's also assume we have L layers, then we have N^2*L weights. With backpropagation, each layer's gradient vector has the size of N^2 entries.

My question in this: The entire MLP is basically a big function containing matrix multiplications and functions of those (like softmax at the end and so on). We are able to calculate the error function by using this function (using relatively simple things like subraction, squaring etc.). This big error function contains the N^2*L weights. I would think we should be able to calculate the gradient of the entire network by deriving it from the error function derived by the N^2*L weights, which would result in a gradient vector of N^2*L entries and update all weights at once with each gradient descend step. Considering that we have far fewer layers than weights per layer, this is a vector space that is not that much larger than the N^2-dimensional gradient vectors that we calculated with backpropagation. So why don't we do this?

I suspect something keeps us from calculating the gradient of the entire MLP - is that so, but what exactly if the MLP is basically a functions which is so well formulated?

(If so, then we could at least use gradient-free optimization methods, and my follow-up questions would be, why we don't and why backpropagation is still so dominant. I read somewhere in the context of Tensorflow, we cannot do without backpropagation, period. But that seems strange to me.)

Answer 1

The main point is that the backpropagation is the efficient way to compute a gradient of the error w.r.t. each weight.

By using backpropagation, you efficiently save computation because you can express gradients of errors in one layer as functions of gradients of the next deeper layer (using the chain rule) instead of computing them from scratch.