Is it possible to give variable sized images as input to a convolutional neural network?

Question

Can we give images with variable size as input to a convolutional neural network for object detection? If possible, how can we do that?

---

But if we try to crop the image, we will be loosing some portion of the image and if we try to resize, then, the clarity of the image will be lost. Does it mean that using inherent network property is the best if image clarity is the main point of consideration?

Answer 1

There are a number of ways to do it. Most of these have already been covered in a number of posts over StackOverflow, Quora and other content websites.

To summarize, most of the techniques listed can be grouped into two classes of solutions, namely,

1. Transformations
2. Inherent Network Property

In transformations, one can look up techniques such as

• Resize, which is the simplest of all the techniques mentioned
• Crop, which can be done as a sliding window or one-time crop with information loss

One can also look into networks that have inherent property to be immune to the size of the input by the virtue of layer behaviour which builds up the network. Examples of this can be found in terms of,

• Fully convolutional networks (FCN), which have no limitations on the input size at all because once the kernel and step sizes are described, the convolution at each layer can generate appropriate dimension outputs according to the corresponding inputs.

• Spatial Pyramid Pooling (SPP), FCNs do not have a fully connected dense layer and hence are agnostic to the image size, but say if one wanted to use dense layer without considering input transformations, then there is a interesting paper that explains the layer in a deep learning network.

References:

1. https://www.quora.com/How-are-variably-shaped-and-sized-images-given-inputs-to-convoluted-neural-networks
2. https://ai.stackexchange.com/questions/2008/how-can-neural-networks-deal-with-varying-input-sizes
3. https://discuss.pytorch.org/t/how-to-create-convnet-for-variable-size-input-dimension-images/1906

P.S. I might have missed citing a few techniques. Not claiming this to be an exhaustive list.

Answer 2

The convolutional layers and pooling layers themselves are independent of the input dimensions. However, the output of the convolutional layers will have different spatial sizes for differently sized images, and this will cause an issue if we have a fully connected layer afterwards (since our fully connected layer requires a fixed size input). There are several solutions to this:

1. Global Pooling: Avoid fully connected layers at the end of the convolutional layers, and instead use pooling (such as Global Average Pooling) to reduce your feature maps from a shape of (N,H,W,C) (before global pool) to shape (N,1,1,C) (after global pool), where:

N = Number of minibatch samples
H = Spatial height of feature map
W = Spatial width of feature map
C = Number of feature maps (channels)

As can be seen, the output dimensionality (N*C) is now independent of the spatial size (H,W) of the feature maps. In case of classification, you can then proceed to use a fully connected layer on top to get the logits for your classes.

2. Variable sized pooling: Use variable sized pooling regions to get the same feature map size for different input sizes.

3. Crop/Resize/Pad input images: You can try to rescale/crop/pad your input images to all have the same shape.

---

In the context of transfer learning, you might want to use differently sized inputs than the original inputs that the model was trained with. Here are some options for doing so:

4. Create new fully connected layers: You can ditch the original fully connected layers completely and initialize a new fully connected layer with the dimensionality that you need, and train it from scratch.

5. Treat the fully connected layer as a convolution: Normally, we reshape the feature maps from (N,H,W,C) to (N,H*W*C) before feeding it to the fully connected layer. But you can also treat the fully connected layer as a convolution with a receptive field of (H,W). Then, you can just convolve this kernel with your feature maps regardless of their size (use zero padding if needed) [http://cs231n.github.io/transfer-learning/ ].

Answer 3

I had to work through this problem today so I thought I'd share what I found that worked. I've found that there were quite a few "this could work in theory" answers and tidbits on the web but less from a practical "here's how you concretely implement this".

To implement this using Tensorflow Keras, I had to do the following. Perhaps someone else can find some of these can be modified, relaxed, or dropped.

1. Set the input of the network to allow for a variable size input using "None" as a placeholder dimension on the input_shape. See Francois Chollet's answer here.
2. Use convolutional layers only until a global pooling operation has occurred (e.g. GlobalMaxPooling2D). Then Dense layers etc. can be used because the size is now fixed.
3. Use a batch size of 1 only. This avoids dealing with mixed sizes within a batch.
4. Write a small custom Sequence that creates batches of size 1 from the list of inputs. I did this to avoid dealing with different sizes inside a single Numpy array.
5. Use Model.fit_generator on your custom Sequence for training and validation. (v.s. Model.fit)
6. For some reason, Model.predict_generator popped even when using the Sequence as above. I had to resort to using Model.predict on individual inputs.

Note that calls to Model.predict did complain about performance - which is unsurprising given the inefficiency of the solution - but it works!

Answer 4

Yes, simply select an appropriate backbone network which doesn't rely on the size of the input image to be some precise value -- most networks satisfy this criteria.