In trying to implement ResNet with bottleneck blocks for myself, I got very confused about the identity function residual blocks with different dimensions. They compared identity, conv projections on multidimensional blocks, conv projections on all blocks. I decided to go with identity as the other don't increase the accuracy substantially but the increase training memory and parameter counts. I noticed the identity is actually an identity of stride 2. (To my understanding, this is essentially max pooling with a kernel of (1,1) and stride (2,2), and concatenating a bunch of zeros). Meaning you lose 3/4s of the identity, and have a new matrix 'identity' of size
(num_filters * 2, n/2, n/2)
and the back half full of zeros. Where n is the 'height' and 'width' dimensions, and num_filters is the number of filters from the previous layer.
E.g. first different-dimensional bottleneck block (omitting batch norm and activation)
Input (256,56, 56) Conv (kernel 1x1, 128 filters, stride 1) (128, 56, 56) Conv (kernel 3x3, 128 filters, stride 2, pad 1) (128, 28, 28) Conv (kernel 1x1, 512 filters, stride 1) (512, 28, 28) Sum (last_conv, confusing_identity)
Hence confusing_identity is of size 512, 28, 28
Wouldn't a max pooling of kernel (3,3) pad (1,1) and stride(2,2) encode more information than this lossy identity?
Also, does any of this really affect training time substantially? (Since both implementations have 0 parameters)