Max Pooling vs Average Pooling for residual/skip connections

Question

I've implemented a CNN with skip connections; some connections skip across residual blocks with no spatial downsampling but some connections skip across blocks that have convolutions with a stride of 2 and therefore the width and height of tensors are halved.

Currently I'm using average pooling for this spatial downsampling, but I'm wondering if there would be an advantage to using max pooling to propagate the highest intensity features.

I looked at the original ResNet paper and it seemed to only go into detail about feature count dimension changes for connections but not spatial dimension changes, so I wonder if there has been any new work in the area comparing the two pooling techniques for res nets.

Answer 1

There is a general debate on whether to use max or average pooling or a conv layer (with stride 2) to downsample.

I would argue that today using conv layers is most prevalent as it simply works best (though not in every case, it requires more data).

For classification often the philosophy is: If we find the pattern, we can classify the object. This philosophy is better captured with max-pooling, as you mostly only care about the strongest occurrence (most features occur only once in some small area), whereas average pooling somehow adds noise as it averages the strongest match with some that are rather weak.

However, (i) max-pooling limits learning (gradients are 0 except for the feature with max activation within the pooling area) and (ii) in particular if you use large pooling width (say 5 or more as in resent in the final layer before the linear layer) the idea of features occur only once is not valid as you average across a very large area.

Answer 2

I am trying to understand what you thought on the application of ResNet. As far as I am concerned, the residual block which contains two type of connections. The first one is the identify block which is just a shortcut connection of input tensor. Another one is the convolution block which needs to be downsampled so that the input tensor can have same shape when it adds to the output of the convolution layers in the residual block.

When to use either of them depends on the condition which typically is between the each hop of residual blocks.

You don't necessarily use average pooling or maxpooling within a residual block since the stride=2 downsamples the shape of input tensor.

On the other hand, you need to use AdaptiveAvgPool() and Flatten() in the final layer.