When creating a neural network, do I update the weights after each run of forward then back propogation? Or do I just keep the random weights and update the Delta variables?

I am looking at slide 8 on these notes:https://d396qusza40orc.cloudfront.net/ml/docs/slides/Lecture9.pdf

It says:

For i = m  
 Set a(1) = x(i)  
 Perform Forward-Propogation  
 Compute delta
 Compute DELTA
 QUESTION: Do I update the Weights that I use in Forward-propogation, or do I 
 use random weights and just keep updating the accumulator 'DELTA'? And if I 
 update the weights, do I set them to DELTA?

For pure stochastic gradient descent, you update the weights after each forward-backward.

When you do the backward phase, you compute the gradients w.r.t. the weights.

You then do the following update:

weights = weights + learningRate * gradWeights

I did not mention more complicated update rules for simplicity.

For batch gradient descent, you accumulate your gradients over the whole batch of samples (i.e. you do a forward-backward pass and do gradWeightsBatch = gradWeightBatch + gradWeight after each backward), and then once you are finished processing the batch, you apply the same update rule:

weights = weights + learningRate * gradWeightsBatch

  • $\begingroup$ Exactly. Each training pattern gives you a weight update. You can perform the weight update either immediately (on-line learning), every N samples by adding up the updates (mini-batch), or summing up the updates over the entire epoch and then applying them (batch-learning). The discussion which would be the best is an expansive topic. $\endgroup$ – PawelP Jun 4 '15 at 10:55
  • $\begingroup$ @smhx I am pretty sure you subtract the learningRate * gradWeights(Batch) $\endgroup$ – Sebastian Nielsen Oct 19 '18 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.