Why does overlapped pooling help reduce overfitting in conv nets?

Question

In the seminal paper on ImageNet classification with deep conv nets by Krizhevsky et al., 2012, the authors talk about overlapped pooling in convolutional neural networks, in Section 3.4.

Pooling layers in CNNs summarize the outputs of neighboring groups of neurons in the same kernel map. Traditionally, the neighborhoods summarized by adjacent pooling units do not overlap (e.g., [17, 11, 4]). To be more precise, a pooling layer can be thought of as consisting of a grid of pooling units spaced s pixels apart, each summarizing a neighborhood of size z × z centered at the location of the pooling unit. If we set s = z, we obtain traditional local pooling as commonly employed in CNNs. If we set s < z, we obtain overlapping pooling. This is what we use throughout our network, with s = 2 and z = 3. This scheme reduces the top-1 and top-5 error rates by 0.4% and 0.3%, respectively, as compared with the non-overlapping scheme s = 2, z = 2, which produces output of equivalent dimensions. We generally observe during training that models with overlapping pooling find it slightly more difficult to overfit.

What is the intuition that overlapped pooling helps reduce over-fitting in conv-nets?

Answer 1

I think it's just that larger pooling windows have lower capacity. For example, if we consider the 1D case, you might imagine you have some features like this:

[0 0 5 0 0 6 0 0 3 0 0 4 0 0]

perhaps generated by some regular grid-like pattern in the original image space. With $z=2$ and $s = 2$ the pooled result is

[0, 5, 6, 0, 3, 4, 0]

and it is still apparent that there is some alternation between high values and low values. but when we increase the window size to 3 we get

[5, 5, 6, 3, 3, 4, 0]

and the grid like pattern is completely smoothed-out and lost.

This is just a contrived example, but a good way to think about it in general is that any large value in a feature map will dominate and mask out all other information within a $z$ by $z$ window after max pooling, so the larger $z$ is, the more information is lost.

Answer 2

I am going to answer this with the pooling example given above with some modifications. Let us say we have three 1D features as given below.

[0 0 5 0 0 6 0 0 3 0 0 4 0 0]

[0 0 0 5 0 6 0 0 0 3 0 4 0 0]

[0 0 5 0 0 6 0 0 3 0 4 0 0 0]

When pooled using z=2 and s=2, all 3 features lead to the same result as obtained above, that is

[0, 5, 6, 0, 3, 4, 0]

However when we use z=3 and s=2, we get the following results respectively

[5, 5, 6, 3, 3, 4, 0]

[0, 5, 6, 0, 3, 4, 0]

[5, 5, 6, 3, 4, 4, 0]

Therefore, with overlapping pooling, we get three different results as opposed to one result when do not use overlapping. This is due to information loss when z=s which in this case leads to reduction in the amount of data available to train the network, i.e from 3 examples to 1 example. The shrinkage in the data size makes the training model overfit.

Answer 3

I think this can be better explained from a digital signal processing point of view.

Intuitively max-pooling is a non-linear sub-sampling operation. Average pooling, on the other hand can be thought as low-pass (averaging) filter followed by sub-sampling. As it has been outlined by Shimao with a nice example, the more the window size is increased, the more information is lost.

Considering first average pooling (which is linear and thus maybe easier to understand) if overlapping is maximum it is in fact just a convolution with a moving average kernel followed by sub-sampling. In other words it is akin to a anti-aliasing (low-pass) filter followed by a decimation operation. If overlapping is not one we are decimating more (just apply the dsp noble identities) thus we are subjected to spatial aliasing as more information is lost and the anti-aliasing filter certainly would not suffice.

Back to max-pooling the reasoning should be similar.

If the pooling regions do not overlap, the pooling regions are disjointed and if that is the case, more information is lost in each pooling layer. If some overlap is allowed the pooling regions overlap with some degree and less spatial information is lost in each layer.

Loss of spatial information by pooling even if is thought to give some degree of spatial invariance to CNNs can be detrimental if abused because it can lead to overfitting as the network will "focus" only on some dominant features; but because the pooling regions are disjointed, it looses quickly any information (in higher layers) of where the feature is located in the image. This obviously can happens for high capacity, deep, models. Letting the pooling regions overlap allows to mitigate for this effect.

Answer 4

A good answer I found on Quora:

Overfitting is not defined over a particular size of dataset. If we have large datasets such as the imagenet, then overfitting could very well be applicable for a large portion of it or a small portion of it. It is simply a phenomenon where a CNN might not learn to extract rich features (more generalizable model), but rather tends to extract features that are only good for classifying certain number of examples in the training set. Excuse my reference to only CNNs, but this argument can be extended to general machine learning.

So, when we have non-overlapping pooling regions, we can see that the spatial information is quickly lost and the network "sees" only the dominant pixel values (winning unit for max pooling for example). This would still provide hierarchical representations, but they would almost always be dominated by the "stronger regions" of an image which then propagate through the network. Effectively, this creates a "bias" in learning which in turn causes an easier potential to overfit.

Let me illustrate this in a different manner. Suppose I take photographs and cover most of it except a bright spot and show them to you, the bright spots all look similar and you learn to identify it. But they could have been from different sources such as a lamp post, headlight of a car, the sun and so on. Unless information from the surrounding is also captured, finer distinguishing cannot be achieved. The overfitting occurs because you almost always see a bright spot and learn to say its a "car" and you turn out to be right since most cars have headlights and you have a lot of photographs of cars having their headlights on, but thats not the only feature you would use to recognise a car and thus would do bad during validation and testing. With overlapping regions, there is less loss of surrounding spatial information. This is why fractional pooling seems even more effective. Note that non-overlapping pooling does not always cause a problem in practice and overlapping pooling regions or fractional pooling only marginally improves results. I had to explain it more dramatically to put forth my point.