Overlap-tile strategy in U-Nets

Question

I was reading the U-Nets paper and there is a mention of some "overlap-tile strategy" in it that I am not quite familiar with. Here is the paragraph from the paper where it has been introduced:

What do they mean by "only us[ing] the valid part of each convolution"? I looked this up and from what I have understood I think they mean that their convolution operations do not involve any padding. Instead of having padded-convolutons to maintain the spatial size of the feature maps, they pad the original image by mirroring the borders and forward the pre-padded image through the network, where it gets downsampled at every convolution and the final output comes out downsampled to the same spatial dimension as the original image. If this strategy really is better than padded-convolutions, why is this not being used everywhere? If not, what is it that makes it worse?

Also, what is the overlap-tile strategy? And how does it allow the "seamless segmentation of arbitrarily large images"? The Figure 2 they are referring to is this, but I am finding it difficult to see what the figure is trying to depict.

Answer 1

On the "overlap-tile strategy" specifically:

The blue box in Fig 2 (left) shows the input to the network. Because they're using valid convolutions, the output is the smaller yellow box (right). Sounds like you understand this part already.

They're trying to show that the image that they want to predict on is bigger than the input to the network (e.g. perhaps the GPU memory is not big enough to hold the whole thing). So they have to run inference several times using different subsets of the input.

On the right side, imagine shifting the yellow box down so that the two squares are right next to each other (bottom side of original square touches top of shifted square). Do that a bunch of times to "tile" your output space. Now, you need a bigger region of the input (blue) for inference. For non-overlapping yellow boxes (in the output) you will need overlapping blue boxes (for the input).

(If its still not clear I can try drawing a picture)

Answer 2

Yeah their description is a bit confusing. I agree with your interpretation.

This paper has 15k citations, so if that strategy is effective, it's probably pretty commonly used. Otherwise, my only guess could be that other techniques in the paper were more important. I'm not familiar with biomedical computer vision research though.

Answer 3

Instead of having padded convolutions to maintain the spatial size of the feature maps, they pad the original image by mirroring the borders and forward the pre-padded image through the network

I agree with @bogovicj 's explanation.

What do they mean by "only us[ing] the valid part of each convolution"?

Based on your understanding of @bogovicj 's explanation, suppose that the 2nd patches under the 1st one within the yellow box, the 2nd one does not need mirror padding for the border between 1st and 2nd ones, just add the left padding.

the upper border of the 2nd patch is inside the 1st one, the author does not make mirror padding for the 2nd patch, and just picks the part from the 1st area. This is the 'valid part of each convolution'.

Thus, the input map is cut into several patches and belongs to the same image, the border they share is the 'valid part of each convolution' because this border is not produced by mirror padding.