Basically, fully convolutional training takes the whole MxM image and produces outputs for all subimages in a single ConvNet forward pass. Patchwise training explicitly crops out the subimages and produces outputs for each subimage in independent forward passes. Therefore, fully convolutional training is usually substantially faster than patchwise training.
So, for fully convolutional training, you make updates like this:
- Input whole MxM image (or multiple images)
- Push through ConvNet -> get an entire map of outputs (maximum size MxM per image, possibly smaller)
- Make updates using the loss of all outputs
Now while this is quite fast, it restricts your training sampling process compared to patchwise training: You are forced to make a lot of updates on the same image (actually, all possible updates for all subimages) during one step of your training. That's why they write that fully convolutional training is only identical to patchwise training, if each receptive field (aka subimage) of an image is contained in a training batch of the patchwise training procedure (for patchwise training, you also could have two of ten possible subimages from image A, three of eight possible subimages from image B, etc. in one batch).
Then, they argue that by not using all outputs during fully convolutional training, you get closer to patchwise training again (since you are not making all possible updates for all subimages of an image in a single training step). However, you waste some of the computation. Also, in Section 4.4/Figure 5, they describe that making all possible updates works just fine and there is no need to ignore some outputs.