What is the memory problem of Shake-Shake regularization?

Question

A ICLR submitted paper develops the idea of Shake-shake regularization by combining it with best known vanila ResNet (Pyramid net) and doing some parameter search. The paper mentions, that Shake-shake regularization has a memory issue.

A indirect hint of that can be found in shake-shake paper:

Interestingly, a key hyperparameter on CIFAR-100 is the batch size which, compared to CIFAR-10, has to be reduced from 128 to 32 if using 2 GPUs.* Without this reduction, the E-E-B network does not produce competitive results. As shown in Table 2, the increased regularization produced by the smaller batch size impacts the training procedure selection and makes S-E-I a slightly better choice.

*https://github.com/facebookresearch/ResNeXt

Can somebody elaborate on that topic?

Answer 1

I'm the author of the Shake-Shake regularization paper.

I'm not sure what the authors of the ShakeDrop paper mean exactly. My guess is that you need a minimum number of channels (e.g. 8) in the first level for the system to converge to SOTA results. If you use Shake-Shake then you might need to start with 2 branches of 8 channels instead of 1 branch of 8. This means that the depth is impacted if you want to keep 22M params. I haven't really tried pushing depth so, for a confirmation, you should ask this question on OpenReview directly.

Wrt to the section you are mentionning in my paper, the point I was making was related to CIFAR-100 which needs more regularization than CIFAR-10. One way to do that is to reduce the batch size as this introduces more stochasticity. This was not a memory problem.