Return to Answer

When the weights are being updated, the incremental change is scaled down by the learning rate parameter. The choice of a small enough gradient descent step size is similar to any other optimization problem.

I'm assuming your query refers to whether we should be:

1. updating the weights of each layer separately and finalise each layer before proceeding to the previous layer, or,
2. update weights concurrently for all layers at the end of a back propagation step.

The back-propagation algorithm calculates the delta with respect to each layer's output vs. ideal output and based on the derivative of the error term assigns the delta of each source connection's in proportion to the current value of the weight. This ensures that even though one round of back-propagation updates all layers at once, but this is done proportionally w.r.t. the current weights.

See this animation for visualizing how weights change during training. You could run a simulation of both methods and observe how the learning progresses.

The biases do not need to be updated for each test case one by one; the value of the biases are always 1 for every test case, but the weight of the bias input of each layer will have to be determined by back-propagation. You can implement a vectorized back-propagation implementation which takes the entire training set and calculates the total error for the current set of weights for each neuron's input including the bias term. This is done for each iteration till the stopping criteria is reached.

When the weights are being updated, the incremental change is scaled down by the learning rate parameter. The choice of a small enough gradient descent step size is similar to any other optimization problem. This should ensure you don't cross the minima by taking too large steps in gradient descent.

I'm assuming your query refers to whether we should be:

1. updating the weights of each layer separately and finalize each layer before proceeding to the previous layer, or,
2. update weights concurrently for all layers at the end of a back propagation step.

The error minimization optimization function used in back-propagation uses all the weights together to calculate the "error" value w.r.t. training set and adjusts them all together in one step of the iteration. If you were to train one layer at a time then it would increase the learning time in proportion to the number of layers of the network.

See this animation for visualizing how weights change during training. You could run a simulation of both methods and observe how the learning progresses.

The biases do not need to be updated for each test case one by one; the value of the biases are always 1 for every test case, but the weight of the bias input of each layer will have to be determined by back-propagation. You can implement a vectorized back-propagation implementation which takes the entire training set and calculates the total error for the current set of weights for each neuron's input including the bias term. This is done for each iteration till the stopping criteria is reached.

When the weights are being updated, the incremental change is scaled down by the learning rate parameter. The choice of a small enough gradient descent step size is similar to any other optimization problem.

I'm assuming your query refers to whether we should be:

1. updating the weights of each layer separately and finalise each layer before proceeding to the previous layer, or,
2. update weights concurrently for all layers at the end of a back propagation step.

The back-propagation algorithm calculates the delta with respect to each layer's output vs. ideal output and based on the derivative of the error term assigns the delta of each source connection's in proportion to the current value of the weight. This ensures that even though one round of back-propagation updates all layers at once, but this is done proportionally w.r.t. the current weights.

See this animation for visualizing how weights change during training. You could run a simulation of both methods and observe how the learning progresses.

The biases do not need to be updated for each test case one by one. You can implement a vectorized back-propagation implementation which takes the entire training set and calculates the total error for the current set of weights for each neuron's input including the bias term. This is done for each iteration till the stopping criteria is reached.

When the weights are being updated, the incremental change is scaled down by the learning rate parameter. The choice of a small enough gradient descent step size is similar to any other optimization problem.

I'm assuming your query refers to whether we should be:

1. updating the weights of each layer separately and finalise each layer before proceeding to the previous layer, or,
2. update weights concurrently for all layers at the end of a back propagation step.

The back-propagation algorithm calculates the delta with respect to each layer's output vs. ideal output and based on the derivative of the error term assigns the delta of each source connection's in proportion to the current value of the weight. This ensures that even though one round of back-propagation updates all layers at once, but this is done proportionally w.r.t. the current weights.

See this animation for visualizing how weights change during training. You could run a simulation of both methods and observe how the learning progresses.

The biases do not need to be updated for each test case one by one; the value of the biases are always 1 for every test case, but the weight of the bias input of each layer will have to be determined by back-propagation. You can implement a vectorized back-propagation implementation which takes the entire training set and calculates the total error for the current set of weights for each neuron's input including the bias term. This is done for each iteration till the stopping criteria is reached.

When the weights are being updated, the incremental change is scaled down by the learning rate parameter. The choice of a small enough gradient descent step size is similar to any other optimization problem.

I'm assuming your query refers to whether we should be:

1. updating the weights of each layer separately, or,
2. update weights concurrently at the end of a back propagation step.

The back-propagation algorithm calculates the delta with respect to each layer's output vs. ideal output and assigns the delta of each source connection's in proportion to the current value of the weight. This ensures that even though one round of back-propagation updates all layers at once, but this is done proportionally w.r.t. the current weights.

See this animation for visualizing how weights change during training.

When the weights are being updated, the incremental change is scaled down by the learning rate parameter. The choice of a small enough gradient descent step size is similar to any other optimization problem.

I'm assuming your query refers to whether we should be:

1. updating the weights of each layer separately and finalise each layer before proceeding to the previous layer, or,
2. update weights concurrently for all layers at the end of a back propagation step.

The back-propagation algorithm calculates the delta with respect to each layer's output vs. ideal output and based on the derivative of the error term assigns the delta of each source connection's in proportion to the current value of the weight. This ensures that even though one round of back-propagation updates all layers at once, but this is done proportionally w.r.t. the current weights.

See this animation for visualizing how weights change during training. You could run a simulation of both methods and observe how the learning progresses.

The biases do not need to be updated for each test case one by one. You can implement a vectorized back-propagation implementation which takes the entire training set and calculates the total error for the current set of weights for each neuron's input including the bias term. This is done for each iteration till the stopping criteria is reached.