I am trying to understand the purpose of the max pooling layers that are insterted between intermediate convolutional layers. As we know, the outputs of convolutional layers are translational equivariant. That is, if a filter has learn to detect a nose, if the nose is shifted the activation in the resulting feature map will also be shifted by the same amount.
So, we can scan the output of Conv(n) layer with the filters from the Conv(n+1) layer and still detect the relevant features. Then, we can use a global pooling operation after the final Conv layer to make our architecture translation invariant. What is the purpose of the Max Pooling layer then?
In the Deep Learning Book, it is stated that:
Invariance to local translation can be a very useful property if we care more about whether some feature is present than exactly where it is.
Can't we achieve this by just using a Global Pooling layer after the final Conv layer?
I add the following image from the Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow from which I can see a potential disadvantage of not using Max Pooling layers:
Suppose that the black pixels (lets say black pixel means 1 and white pixel means 0) on the output of the Max Pooling correspond to some detected feature. Then, the output of next Conv layer, assuming a filter with values and a stride of 2: $$ \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix} $$
Conv output (assuming left Max Pooling) $$\begin{bmatrix} 2 & 2 \\ 2 & 2 \\ \end{bmatrix} $$
Conv output (assuming mid Max Pooling) $$\begin{bmatrix} 2 & 2 \\ 2 & 2 \\ \end{bmatrix} $$
Conv output (assuming right Max Pooling) $$\begin{bmatrix} 0 & 4 \\ 0 & 4 \\ \end{bmatrix} $$
Intuitively, the problem I see is that by not being translation invariant the following layers after this Conv layer will receive different outputs and this will make their job harder during training since they have to readjust the values of their filter weights depending on where exactly the feature was detected. Is this intuition correct?
