Neural Networks back propagation

Question

I have gone through neural networks and have understood the derivation for back propagation almost perfectly (finally!). However, I had a small doubt. We are updating all the weights simultaneously, so what is the guarantee that they lead to a smaller cost? If the weights are updated one by one, it would definitely lead to a lower cost and it would be similar to linear regression. But if you update all the weights simultaneously, might we not cross the minima? Also, do we update the biases like we update the weights after each forward propagation and back propagation of each test case?

Answer 1

When the weights are being updated, the incremental change is scaled down by the learning rate parameter. The choice of a small enough gradient descent step size is similar to any other optimization problem. This should ensure you don't cross the minima by taking too large steps in gradient descent.

I'm assuming your query refers to whether we should be:

1. updating the weights of each layer separately and finalize each layer before proceeding to the previous layer, or,
2. update weights concurrently for all layers at the end of a back propagation step.

The error minimization optimization function used in back-propagation uses all the weights together to calculate the "error" value w.r.t. training set and adjusts them all together in one step of the iteration. If you were to train one layer at a time then it would increase the learning time in proportion to the number of layers of the network.

See this animation for visualizing how weights change during training. You could run a simulation of both methods and observe how the learning progresses.

The biases do not need to be updated for each test case one by one; the value of the biases are always 1 for every test case, but the weight of the bias input of each layer will have to be determined by back-propagation. You can implement a vectorized back-propagation implementation which takes the entire training set and calculates the total error for the current set of weights for each neuron's input including the bias term. This is done for each iteration till the stopping criteria is reached.

Answer 2

As I can understand, basically the question is why do we update weights in all layers simultaneously, and not layer by layer? The layers seems interconnected, later layers depend on previous, so it seems more right to do it step by step, right? Yes, you can do this but it is not necessary, because variables W1, W2, W3, etc. independent from each other!

If you change the W1, then W2 will not change at all. So you can apply a partial derivative rule as in classic examples of function of two independent variables.

Update: You probably also be interested in the multivariable chain rule, which can give you the answer of how much the cost function will change if you change a little both W1 and W2 simultaneously.