I am trying to train a large model (deep net using caffe) using stochastic gradient descent (SGD).
The problem is I am constraint by my GPU memory capacity and thus cannot process large mini-batches for each stochastic gradient estimation.

How can I overcome this instability in my training?

One thought I had was to use momentum, and set it to a higher value than the default is usually set to. Is this a valid strategy?

For those of you who are happen to use Caffe, it might be interesting to know that caffe has already implemented the gradient accumulation across mini-batches (as suggested by Indie Al). You simply need to define iter_size in the 'solver.prototxt'.

This can also be done in pytorch. See this post for example.

  • $\begingroup$ Please do not cross-post. Pick the site where you want to post & delete the other thread. $\endgroup$ Mar 15, 2016 at 8:58
  • $\begingroup$ @gung currently, I get no responses on both sites. Once I get a response on any of the sites, I'll remove from the other. $\endgroup$
    – Shai
    Mar 15, 2016 at 8:59
  • $\begingroup$ @gung getting a pulse here, I removed the original question for SO. $\endgroup$
    – Shai
    Mar 15, 2016 at 15:54
  • $\begingroup$ In the original paper introducing U-Net, the authors mention that they reduced the batch size to 1 (so they went from mini-batch GD to SGD) and compensated by adopting a momentum of 0.99. They got SOTA results, but it's hard to determine what role this decision played. $\endgroup$
    – David Cian
    Feb 11, 2021 at 13:39
  • $\begingroup$ @DavidCian on the other hand there's this paper that suggest a linear relation between the learning rate and the batch size... looks more like dark arts than science... :O $\endgroup$
    – Shai
    Feb 11, 2021 at 13:46

2 Answers 2


With a small batchsize, I believe the SGD descent direction becomes a very noisy estimate of the "true" descent direction (i.e. if we evaluated it on the entire training set). With a small batchsize, I am not sure how much increasing the momentum would help as it would be accumulating momentum in very noisy directions. But I could be wrong, maybe your optimization problem is well-posed enough where this could work.

If you aren't gunning for "state of the art" results, one option you have for natural image data is to resize the images. I actually think that modulo chasing down elite benchmarking performance, natural images have a lot of scale invariant properties and a lot of their semantic features are fairly robust under reasonable scaling transformations. This would alleviate some of the GPU memory and allow you to increase your batchsize and your SGD descent directions would be better estimates of the descent directions.

If you are dealing with a separable loss function like negative log likelihood, we can exploit the fact that the gradient of a large batch is merely the sum/average of the gradients of its constituent sub-batches. For example if our batchsize is $B$, we can compute gradients of a super batchsize $BK$ by iterating through the batches as usual, computing each batch gradient, but instead of updating the weights, we cache each gradient into a running sum or average. If we average appropriately, we will be computing the exact gradient for the $BK$ sized super batch. We then perform the weight update after each $K$-th batch has been processed.

We will be exactly computing the $BK$ batch gradients by serializing the computation as described above. There is minimal extra computational or memory overhead, one only needs to modify the minibatch iterator to include the super batch serialization and gradient cache.

  • $\begingroup$ I already scaled down whatever inputs I was able to scale... It's down to mini batch size now... $\endgroup$
    – Shai
    Mar 15, 2016 at 15:11
  • $\begingroup$ thank you for your answer. The thing is I really aim at getting an expert opinion on the ability of momentum to work as a "gradient noise reduction" component $\endgroup$
    – Shai
    Mar 15, 2016 at 15:15
  • $\begingroup$ Ah I see, I have updated my answer to suggest another possibility then. I haven't tried it out, but maybe it can work for you? Hope it helps, even if to figure what what doesn't work. $\endgroup$
    – Indie AI
    Mar 15, 2016 at 16:27
  • $\begingroup$ it's an interesting approach, but in many cases you need the "forward" information in order to compute the "backward" gradient. Therefore, you need to "re-forward" all the mini-batches you want to average in order to estimate the gradient - this is not feasible in terms of training time - I'd move to CPU and get the same speed with larger batches. $\endgroup$
    – Shai
    Mar 15, 2016 at 16:35
  • 1
    $\begingroup$ Okay, it makes more sense now. Although it does sounds a lot like increased momentum... See the last paragraph here: "[momentum] determines for how many iterations the previous gradients are incorporated into the current update" $\endgroup$
    – Shai
    Mar 15, 2016 at 16:49

Recently I came across an interesting work:

Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le, Don't Decay the Learning Rate, Increase the Batch Size (ICLR 2018).

This works shows a direct link between batch size and learning rate. Specifically, decreasing learning rate has the same effect as increasing batch size and vice versa.
Taking their conclusion to extreme, one might consider decreasing the batch size and compensating for it by increasing the learning rate.
I haven't actually tried it yet, though.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.