Neural Network Process Question - Updating weights after each training set

Question

When creating a neural network, do I update the weights after each run of forward then back propogation? Or do I just keep the random weights and update the Delta variables?

I am looking at slide 8 on these notes:https://d396qusza40orc.cloudfront.net/ml/docs/slides/Lecture9.pdf

It says:

For i = m  
 Set a(1) = x(i)  
 Perform Forward-Propogation  
 Compute delta
 Compute DELTA
 QUESTION: Do I update the Weights that I use in Forward-propogation, or do I 
 use random weights and just keep updating the accumulator 'DELTA'? And if I 
 update the weights, do I set them to DELTA?

Answer 1

For pure stochastic gradient descent, you update the weights after each forward-backward.

When you do the backward phase, you compute the gradients w.r.t. the weights.

You then do the following update:

weights = weights + learningRate * gradWeights

I did not mention more complicated update rules for simplicity.

For batch gradient descent, you accumulate your gradients over the whole batch of samples (i.e. you do a forward-backward pass and do gradWeightsBatch = gradWeightBatch + gradWeight after each backward), and then once you are finished processing the batch, you apply the same update rule:

weights = weights + learningRate * gradWeightsBatch