A ICLR submitted paper develops the idea of Shake-shake regularization by combining it with best known vanila ResNet (Pyramid net) and doing some parameter search. The paper mentions, that Shake-shake regularization has a memory issue.
A indirect hint of that can be found in shake-shake paper:
Interestingly, a key hyperparameter on CIFAR-100 is the batch size which, compared to CIFAR-10, has to be reduced from 128 to 32 if using 2 GPUs.* Without this reduction, the E-E-B network does not produce competitive results. As shown in Table 2, the increased regularization produced by the smaller batch size impacts the training procedure selection and makes S-E-I a slightly better choice.
Can somebody elaborate on that topic?