2
$\begingroup$

A ICLR submitted paper develops the idea of Shake-shake regularization by combining it with best known vanila ResNet (Pyramid net) and doing some parameter search. The paper mentions, that Shake-shake regularization has a memory issue.

A indirect hint of that can be found in shake-shake paper:

Interestingly, a key hyperparameter on CIFAR-100 is the batch size which, compared to CIFAR-10, has to be reduced from 128 to 32 if using 2 GPUs.* Without this reduction, the E-E-B network does not produce competitive results. As shown in Table 2, the increased regularization produced by the smaller batch size impacts the training procedure selection and makes S-E-I a slightly better choice.

*https://github.com/facebookresearch/ResNeXt

Can somebody elaborate on that topic?

$\endgroup$
2
$\begingroup$

I'm the author of the Shake-Shake regularization paper.

I'm not sure what the authors of the ShakeDrop paper mean exactly. My guess is that you need a minimum number of channels (e.g. 8) in the first level for the system to converge to SOTA results. If you use Shake-Shake then you might need to start with 2 branches of 8 channels instead of 1 branch of 8. This means that the depth is impacted if you want to keep 22M params. I haven't really tried pushing depth so, for a confirmation, you should ask this question on OpenReview directly.

Wrt to the section you are mentionning in my paper, the point I was making was related to CIFAR-100 which needs more regularization than CIFAR-10. One way to do that is to reduce the batch size as this introduces more stochasticity. This was not a memory problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.