I have gone through neural networks and have understood the derivation for back propagation almost perfectly (finally!). However, I had a small doubt. We are updating all the weights simultaneously, so what is the guarantee that they lead to a smaller cost? If the weights are updated one by one, it would definitely lead to a lower cost and it would be similar to linear regression. But if you update all the weights simultaneously, might we not cross the minima? Also, do we update the biases like we update the weights after each forward propagation and back propagation of each test case?

  • $\begingroup$ This question is currently too broad - you're really asking a whole bunch of questions and they aren't particularly closely related. Our Q&A format works better by asking a very specific question and getting specific answers. Have a look at our guide on asking a good question. $\endgroup$
    – Silverfish
    May 22, 2016 at 8:30
  • $\begingroup$ Hey, thanks for the advice! I edited the question after going through the guide. $\endgroup$
    – RaviTej310
    May 22, 2016 at 8:51
  • $\begingroup$ If the learning rate is sufficiently small, the weight update will ALWAYS reduce the value of the objective function, provided the gradient is non-zero. $\endgroup$
    – user18764
    May 2, 2018 at 20:20

2 Answers 2


When the weights are being updated, the incremental change is scaled down by the learning rate parameter. The choice of a small enough gradient descent step size is similar to any other optimization problem. This should ensure you don't cross the minima by taking too large steps in gradient descent.

I'm assuming your query refers to whether we should be:

  1. updating the weights of each layer separately and finalize each layer before proceeding to the previous layer, or,
  2. update weights concurrently for all layers at the end of a back propagation step.

The error minimization optimization function used in back-propagation uses all the weights together to calculate the "error" value w.r.t. training set and adjusts them all together in one step of the iteration. If you were to train one layer at a time then it would increase the learning time in proportion to the number of layers of the network.

See this animation for visualizing how weights change during training. You could run a simulation of both methods and observe how the learning progresses.

The biases do not need to be updated for each test case one by one; the value of the biases are always 1 for every test case, but the weight of the bias input of each layer will have to be determined by back-propagation. You can implement a vectorized back-propagation implementation which takes the entire training set and calculates the total error for the current set of weights for each neuron's input including the bias term. This is done for each iteration till the stopping criteria is reached.


As I can understand, basically the question is why do we update weights in all layers simultaneously, and not layer by layer? The layers seems interconnected, later layers depend on previous, so it seems more right to do it step by step, right? Yes, you can do this but it is not necessary, because variables W1, W2, W3, etc. independent from each other!

If you change the W1, then W2 will not change at all. So you can apply a partial derivative rule as in classic examples of function of two independent variables.

Update: You probably also be interested in the multivariable chain rule, which can give you the answer of how much the cost function will change if you change a little both W1 and W2 simultaneously.

  • $\begingroup$ please add a reference for your link if possible (a link can die) $\endgroup$
    – Antoine
    May 2, 2018 at 10:42
  • $\begingroup$ @Antoine I switched link to point to the archive.org snapshot. I afraid that this page don't have any official title. $\endgroup$ May 2, 2018 at 11:29
  • $\begingroup$ I'm not sure whether this is common practice on this site, but this sure does address the link dying problem :) $\endgroup$
    – Antoine
    May 2, 2018 at 11:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.