6
$\begingroup$

In the seminal paper on ImageNet classification with deep conv nets by Krizhevsky et al., 2012, the authors talk about overlapped pooling in convolutional neural networks, in Section 3.4.

Pooling layers in CNNs summarize the outputs of neighboring groups of neurons in the same kernel map. Traditionally, the neighborhoods summarized by adjacent pooling units do not overlap (e.g., [17, 11, 4]). To be more precise, a pooling layer can be thought of as consisting of a grid of pooling units spaced s pixels apart, each summarizing a neighborhood of size z × z centered at the location of the pooling unit. If we set s = z, we obtain traditional local pooling as commonly employed in CNNs. If we set s < z, we obtain overlapping pooling. This is what we use throughout our network, with s = 2 and z = 3. This scheme reduces the top-1 and top-5 error rates by 0.4% and 0.3%, respectively, as compared with the non-overlapping scheme s = 2, z = 2, which produces output of equivalent dimensions. We generally observe during training that models with overlapping pooling find it slightly more difficult to overfit.

What is the intuition that overlapped pooling helps reduce over-fitting in conv-nets?

$\endgroup$
1
$\begingroup$

I think it's just that larger pooling windows have lower capacity. For example, if we consider the 1D case, you might imagine you have some features like this:

[0 0 5 0 0 6 0 0 3 0 0 4 0 0]

perhaps generated by some regular grid-like pattern in the original image space. With $z=2$ and $s = 2$ the pooled result is

[0, 5, 6, 0, 3, 4, 0]

and it is still apparent that there is some alternation between high values and low values. but when we increase the window size to 3 we get

[5, 5, 6, 3, 3, 4, 0]

and the grid like pattern is completely smoothed-out and lost.

This is just a contrived example, but a good way to think about it in general is that any large value in a feature map will dominate and mask out all other information within a $z$ by $z$ window after max pooling, so the larger $z$ is, the more information is lost.

$\endgroup$
0
$\begingroup$

I think this can be better explained from a digital signal processing point of view.

Intuitively max-pooling is a non-linear sub-sampling operation. Average pooling, on the other hand can be thought as low-pass (averaging) filter followed by sub-sampling. As it has been outlined by Shimao with a nice example, the more the window size is increased, the more information is lost.

Considering first average pooling (which is linear and thus maybe easier to understand) if overlapping is maximum it is in fact just a convolution with a moving average kernel followed by sub-sampling. In other words it is akin to a anti-aliasing (low-pass) filter followed by a decimation operation. If overlapping is not one we are decimating more (just apply the dsp noble identities) thus we are subjected to spatial aliasing as more information is lost and the anti-aliasing filter certainly would not suffice.

Back to max-pooling the reasoning should be similar.

If the pooling regions do not overlap, the pooling regions are disjointed and if that is the case, more information is lost in each pooling layer. If some overlap is allowed the pooling regions overlap with some degree and less spatial information is lost in each layer.

Loss of spatial information by pooling even if is thought to give some degree of spatial invariance to CNNs can be detrimental if abused because it can lead to overfitting as the network will "focus" only on some dominant features; but because the pooling regions are disjointed, it looses quickly any information (in higher layers) of where the feature is located in the image. This obviously can happens for high capacity, deep, models. Letting the pooling regions overlap allows to mitigate for this effect.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.