Should I use the whole dataset in the forward pass when doing minibatch gradient descent?

Question

I've implemented the following algorithm. For each minibatch:

• Compute the gradient using the mini-batch sample
• Update the parameters
• Update the hidden layers. If $\Gamma_L$ are the new parameters at the $L$th layer, $\mathbf{X}$ is the input data, and $a()$ is the element-wise activation function:

$$ \hat y = a(a(...a(\mathbf{X}\Gamma_1)\Gamma_2) ...)\Gamma_l $$

I need $\hat y$ to compute the working residual, in order to compute the first step in the chain rule to get the gradient for the subsequent backprop step. But using the whole dataset in the forward pass is slow when sample size is big, and probably impossible when it is huge.

Up until now I've been working with modest samples, and this problem hasn't occurred to me.

Is it standard practice to only update $\hat y_{i \in minibatch}$?? By computing $$ \hat y_{i \in MB} = a(a(...a(\mathbf{X}_{i \in MB}\Gamma_1)\Gamma_2) ...)\Gamma_l $$ ??

That way each observation's estimate will get updated once per epoch. But this seems weird, because the working residuals for each backward pass would no longer correspond to the parameters used to compute the current gradient. In other words, the parameters would be updated, but the working residual wouldn't.

I guess I had been doing halfway minibatching -- only minibatching for the backward pass. Could someone confirm what is standard practice for the forward pass?

Maybe I've just been doing it backwards? Is it standard to first update $\hat y$ for a minibatch, then compute the gradient?

Answer 1

Tl;dr: the standard procedure is to use a minibatch it for both forward and backward pass.

What you're trying to do is to run a stochastic gradient descent on your neural network. The step are similar to what you've written, but differ in an important way:

1. Sample a small number of data points.
2. Run forward pass on a minibatch.
3. Run backward pass on a minibatch.

I'm not sure if I correctly understood the part about "parameters get updated, but the working residual doesn't". However, I'll still try to answer that part :)

Even though it may seem weird, there is a solid reasoning behind that. Namely, even though you never get the true gradient vector, you get a vector that is on average equal to the gradient. The hope is that you'll go in the "right" way most of the time and (because you can run a lot of steps) you'll eventually get in the optimal point.

Hope that helps :)