Let say we have two affine layers with
y = ReLU(w*x+b). We have two variables which are storing the gradient value of each layer (
Let say during backward pass we have computed the gradient of the second layer (for example this is
CurGrad variable) and so we want to update
Grad1 variable. Should we overwrite current
Grad1 value or should we just add new Gradient to current gradient value. So should we use
Grad1 = CurGrad or
Grad1 += CurGrad? In the Andrej Karpathy: Hacker's guide to Neural Networks I see
Grad1 += CurGrad approach. And that looks for me as some form of linear filter. So this should we more robust but also more inertial approach. If will go to a wrong direction, it would be harder to correct direction fast. I have tested both variants - both works nearly they same in my toy test.
Second question how is this related to batch/SGD/minibatch approaches?
As I understand in SGD and batch we will never set
Grad1 to zero and in minibatch we will set Grad1 to zero after every mini batch. Am I right?