Does the property of equivariance to translation of convolution layers help to learn translation-invariant features? [duplicate]

Question

In some texts, people mention that the reason why convolutional neural networks are able to learn translation-invariant features are related to the property that convolution layers are equivariant to translation, which means the convolution of the inputs being shifted is the same as the shifted output of the convolution of the original inputs.

But I am a little bit confused about this, since the output signals of the convolution layer gets still shifted anyway ("equivariant" is not "invariant", the latter case not only preserves the magnitude, but also preserve the location of the output signal), so there should be some change in the later outputs unless the the coefficients of the connection to any of the nodes in this layer to be the same.

I also read that the pooling layer may help with this translation issue, but it is only approximately invariant to very small local translation. What if I move the image of a cat from one corner of the image to another corner? How is CNN able to realize that it is still a cat but only the position is shifted?

Answer 1

What causes convolutional neural networks to be somewhat translation invariant is the max pooling. Each neuron has a receptive field in the original image. For example, if you have two convolutional layers with stride 1 and one 2x2 max pooling step in between,

That is, input image --> C3x3/1 --> M2x2/2 --> C3x3/1 --> output feature map,

then each neuron in the output feature map sees 8x8 patches in the original image, i.e. has a 8x8 receptive field. That neuron gets excited by stuff that happens anywhere in this 8x8 region (ignoring border effects) because the spatial information was lost in the max pooling step. If you add more max pooling steps to the network you will increase this receptive field.

Typically, in the last few layers, densely connected layers are used, which combine the information from the different receptive fields. There, different regions of the image are connected with different weights, so it does matter where the information came from.

For example, in a face recognition software you might want to abstract the information a bit through max pooling, but not too much, because the information how the different image components (eyes, nose etc.) are spatially related is important.

Or, expanding on the example you gave. Imagine you were to train a network with images of cats and dogs in which the animals only ever appear in the upper left corner. Furthermore, you design the network such that the receptive field of your last feature map before the fully connected layer is a quarter of the input image. Then the classifier would not be able to recognise a cat or a dog in the lower right corner. The weights in the fully connected layer connecting to that part of the image would never have learned anything.

Lastly, you can make your network so deep that the receptive fields of the last layer before the fully connected layers, covers the whole image. In that case, anything in the input image can excite any neuron in the last feature map.

Answer 2

I think the equivariance property does carry over consecutive convolutional layers if you had a chain of ConvNet without anything in between. But in practice, you have a relu or a pool layer and so that equivariant property doesn't hold across layers.

For Pooling, I think it only helps with small translations in the input, keeping the output fairly constant (in the case of max-pool for example) and allowing the layer above to better learn that representation. I don't think Pooling helps with large translation like the one where a cat is moved from one corner to the other. Convolution rather would help with that, since it will make that signal more obvious to the later layers by giving a translated and proportionate increase in the output, for both pictures.