1
$\begingroup$

Let say we have two affine layers with y = ReLU(w*x+b). We have two variables which are storing the gradient value of each layer (Grad1 and Grad2).

Let say during backward pass we have computed the gradient of the second layer (for example this is CurGrad variable) and so we want to update Grad1 variable. Should we overwrite current Grad1 value or should we just add new Gradient to current gradient value. So should we use Grad1 = CurGrad or Grad1 += CurGrad? In the Andrej Karpathy: Hacker's guide to Neural Networks I see Grad1 += CurGrad approach. And that looks for me as some form of linear filter. So this should we more robust but also more inertial approach. If will go to a wrong direction, it would be harder to correct direction fast. I have tested both variants - both works nearly they same in my toy test.

Second question how is this related to batch/SGD/minibatch approaches? As I understand in SGD and batch we will never set Grad1 to zero and in minibatch we will set Grad1 to zero after every mini batch. Am I right?

$\endgroup$
1
  • $\begingroup$ This is really a more general question about how numerical optimization works and has no content that is not general to optimization strategies. I'm afraid that these kinds of questions are the inevitable result of being satisfied with books entitled "Hacker's guide to..." because the premise is that all you need to know how to do is code and that any mathematics are unimportant. This could not be further from the truth. $\endgroup$ – Sycorax Nov 14 '16 at 23:35
2
$\begingroup$

should we use Grad1 = CurGrad or Grad1 += CurGrad?

That depends on the parameter update strategy. Some accumulate gradients (e.g., Adadelta), some do not (e.g., vanilla SGD).

See http://cs231n.github.io/neural-networks-3/ or http://sebastianruder.com/optimizing-gradient-descent/ for the most commonly used parameter update strategies.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.