I've implemented the following algorithm. For each minibatch:

  • Compute the gradient using the mini-batch sample
  • Update the parameters
  • Update the hidden layers. If $\Gamma_L$ are the new parameters at the $L$th layer, $\mathbf{X}$ is the input data, and $a()$ is the element-wise activation function:

$$ \hat y = a(a(...a(\mathbf{X}\Gamma_1)\Gamma_2) ...)\Gamma_l $$

I need $\hat y$ to compute the working residual, in order to compute the first step in the chain rule to get the gradient for the subsequent backprop step. But using the whole dataset in the forward pass is slow when sample size is big, and probably impossible when it is huge.

Up until now I've been working with modest samples, and this problem hasn't occurred to me.

Is it standard practice to only update $\hat y_{i \in minibatch}$?? By computing $$ \hat y_{i \in MB} = a(a(...a(\mathbf{X}_{i \in MB}\Gamma_1)\Gamma_2) ...)\Gamma_l $$ ??

That way each observation's estimate will get updated once per epoch. But this seems weird, because the working residuals for each backward pass would no longer correspond to the parameters used to compute the current gradient. In other words, the parameters would be updated, but the working residual wouldn't.

I guess I had been doing halfway minibatching -- only minibatching for the backward pass. Could someone confirm what is standard practice for the forward pass?

Maybe I've just been doing it backwards? Is it standard to first update $\hat y$ for a minibatch, then compute the gradient?


1 Answer 1


Tl;dr: the standard procedure is to use a minibatch it for both forward and backward pass.

What you're trying to do is to run a stochastic gradient descent on your neural network. The step are similar to what you've written, but differ in an important way:

  1. Sample a small number of data points.
  2. Run forward pass on a minibatch.
  3. Run backward pass on a minibatch.

I'm not sure if I correctly understood the part about "parameters get updated, but the working residual doesn't". However, I'll still try to answer that part :)

Even though it may seem weird, there is a solid reasoning behind that. Namely, even though you never get the true gradient vector, you get a vector that is on average equal to the gradient. The hope is that you'll go in the "right" way most of the time and (because you can run a lot of steps) you'll eventually get in the optimal point.

Hope that helps :)

  • $\begingroup$ Yeah. I had been doing it backwards. This has no consequence in full batch mode. But in minibatch, doing it backwards means that the residual $y-\hat y$ -- which is the first term in the gradient for a regression problem -- doesn't reflect the last update of the parameters, because $\hat y$ is a function of the layers which haven't been updated yet to reflect the new parameters. $\endgroup$ Mar 1, 2018 at 17:50
  • $\begingroup$ @generic_user Not sure if I understand you correctly-- shouldn't the first term be like "model output - ground truth labels"? $\endgroup$ Mar 1, 2018 at 18:10
  • $\begingroup$ Right. Same thing. $\endgroup$ Mar 1, 2018 at 18:57
  • $\begingroup$ @generic_user Well I don't see any problems in that case. During SGD, you iteratively evaluate gradient (over a minibatch) w.r.t. your weights and update weights using the gradient you computed just before. Your parameters always reflect enough changes for learning a network. $\endgroup$ Mar 2, 2018 at 5:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.