8
$\begingroup$

In the paper of fully convolutional neural network, the authors mention both patch wise training and fully convolutional training.

My understanding for the training set construction is as follows:

Given an M*M image, extract sub-images with N*N, where (N<M). The selected sub-images are overlapped with eath other. For each batch in the training process, it can include all the sub-images for a given image or multiple images.

Is my understanding correct? Then what are the difference between patch-wise training and fully convolutional training? I include the related section as a reference.

enter image description here

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
9
$\begingroup$

Basically, fully convolutional training takes the whole MxM image and produces outputs for all subimages in a single ConvNet forward pass. Patchwise training explicitly crops out the subimages and produces outputs for each subimage in independent forward passes. Therefore, fully convolutional training is usually substantially faster than patchwise training.

So, for fully convolutional training, you make updates like this:

  1. Input whole MxM image (or multiple images)
  2. Push through ConvNet -> get an entire map of outputs (maximum size MxM per image, possibly smaller)
  3. Make updates using the loss of all outputs

Now while this is quite fast, it restricts your training sampling process compared to patchwise training: You are forced to make a lot of updates on the same image (actually, all possible updates for all subimages) during one step of your training. That's why they write that fully convolutional training is only identical to patchwise training, if each receptive field (aka subimage) of an image is contained in a training batch of the patchwise training procedure (for patchwise training, you also could have two of ten possible subimages from image A, three of eight possible subimages from image B, etc. in one batch). Then, they argue that by not using all outputs during fully convolutional training, you get closer to patchwise training again (since you are not making all possible updates for all subimages of an image in a single training step). However, you waste some of the computation. Also, in Section 4.4/Figure 5, they describe that making all possible updates works just fine and there is no need to ignore some outputs.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Hi robintibor, thanks for the reply. You mention that “input whole MxM image (or multiple images)”. Do you mean that feed the whole image into network without creating sub-images before feeding the network? In practice, if the image size is very big, we generally have to create sub-images at first. Is my understanding correct? $\endgroup$
    – user3125
    Mar 19, 2017 at 22:22
  • $\begingroup$ Hi @user3125 yes feed in the whole image, or typically even multiple images in one batch. I guess it rarely happens that a 2d-image is too big to push through the network memorywise, rather maybe you have to reduce your batch size, i.e. the number of (complete, not sub-) images you feed to the network at the same time. $\endgroup$
    – robintibor
    Mar 20, 2017 at 11:19
  • $\begingroup$ Hi robintibor, thanks for the clarification. My scenario is that we have limited number of images(200~300) with large pixel size (980*980). Therefore, I am thinking of creating large number of small images for training purposes. $\endgroup$
    – user3125
    Mar 20, 2017 at 14:40
  • $\begingroup$ Ok I see. For 980*980 dimensional images and three color channels you should have about 11 MB per image assuming float32=4 byte per pixel: (980 * 980 * 3 * 4 Bytes) / (1024 ^ 2.0) = 10.9909 MB. So several images should easily fit onto a regular GPU, but then your network structure (number of filters, number of layers, when you are downsampling etc.) determines how much memory is used for an entire forward-backward pass. I suggest try pushing entire images through the network and increasing the batch size (number of images processed at same time) until it crashes :) $\endgroup$
    – robintibor
    Mar 20, 2017 at 17:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.