51
$\begingroup$

Can we give images with variable size as input to a convolutional neural network for object detection? If possible, how can we do that?


But if we try to crop the image, we will be loosing some portion of the image and if we try to resize, then, the clarity of the image will be lost. Does it mean that using inherent network property is the best if image clarity is the main point of consideration?

$\endgroup$

4 Answers 4

25
$\begingroup$

There are a number of ways to do it. Most of these have already been covered in a number of posts over StackOverflow, Quora and other content websites.

To summarize, most of the techniques listed can be grouped into two classes of solutions, namely,

  1. Transformations
  2. Inherent Network Property

In transformations, one can look up techniques such as

  • Resize, which is the simplest of all the techniques mentioned
  • Crop, which can be done as a sliding window or one-time crop with information loss

One can also look into networks that have inherent property to be immune to the size of the input by the virtue of layer behaviour which builds up the network. Examples of this can be found in terms of,

  • Fully convolutional networks (FCN), which have no limitations on the input size at all because once the kernel and step sizes are described, the convolution at each layer can generate appropriate dimension outputs according to the corresponding inputs.

  • Spatial Pyramid Pooling (SPP), FCNs do not have a fully connected dense layer and hence are agnostic to the image size, but say if one wanted to use dense layer without considering input transformations, then there is a interesting paper that explains the layer in a deep learning network.

References:

  1. https://www.quora.com/How-are-variably-shaped-and-sized-images-given-inputs-to-convoluted-neural-networks
  2. https://ai.stackexchange.com/questions/2008/how-can-neural-networks-deal-with-varying-input-sizes
  3. https://discuss.pytorch.org/t/how-to-create-convnet-for-variable-size-input-dimension-images/1906

P.S. I might have missed citing a few techniques. Not claiming this to be an exhaustive list.

$\endgroup$
2
  • 3
    $\begingroup$ This sounds ok in theory, but doesn't work on Tensorflow. Does anyone have any implementations for it? $\endgroup$
    – Hossein
    Jul 22, 2019 at 15:37
  • 2
    $\begingroup$ @Hossein I also ran into some issues in practical implementation, but I got a variable size CNN working in Tensorflow Keras 2.x today with some limitations. I have posted an outline of the approach in my answer with regards to some of the practical details. Best of luck! $\endgroup$
    – J Trana
    Jan 17, 2020 at 6:14
24
$\begingroup$

The convolutional layers and pooling layers themselves are independent of the input dimensions. However, the output of the convolutional layers will have different spatial sizes for differently sized images, and this will cause an issue if we have a fully connected layer afterwards (since our fully connected layer requires a fixed size input). There are several solutions to this:

1. Global Pooling: Avoid fully connected layers at the end of the convolutional layers, and instead use pooling (such as Global Average Pooling) to reduce your feature maps from a shape of (N,H,W,C) (before global pool) to shape (N,1,1,C) (after global pool), where:

N = Number of minibatch samples
H = Spatial height of feature map
W = Spatial width of feature map
C = Number of feature maps (channels)

As can be seen, the output dimensionality (N*C) is now independent of the spatial size (H,W) of the feature maps. In case of classification, you can then proceed to use a fully connected layer on top to get the logits for your classes.

2. Variable sized pooling: Use variable sized pooling regions to get the same feature map size for different input sizes.

3. Crop/Resize/Pad input images: You can try to rescale/crop/pad your input images to all have the same shape.


In the context of transfer learning, you might want to use differently sized inputs than the original inputs that the model was trained with. Here are some options for doing so:

4. Create new fully connected layers: You can ditch the original fully connected layers completely and initialize a new fully connected layer with the dimensionality that you need, and train it from scratch.

5. Treat the fully connected layer as a convolution: Normally, we reshape the feature maps from (N,H,W,C) to (N,H*W*C) before feeding it to the fully connected layer. But you can also treat the fully connected layer as a convolution with a receptive field of (H,W). Then, you can just convolve this kernel with your feature maps regardless of their size (use zero padding if needed) [http://cs231n.github.io/transfer-learning/ ].

$\endgroup$
1
  • 3
    $\begingroup$ This should be the accepted answer. $\endgroup$
    – Hephaestus
    Nov 30, 2020 at 21:05
9
$\begingroup$

I had to work through this problem today so I thought I'd share what I found that worked. I've found that there were quite a few "this could work in theory" answers and tidbits on the web but less from a practical "here's how you concretely implement this".

To implement this using Tensorflow Keras, I had to do the following. Perhaps someone else can find some of these can be modified, relaxed, or dropped.

  1. Set the input of the network to allow for a variable size input using "None" as a placeholder dimension on the input_shape. See Francois Chollet's answer here.
  2. Use convolutional layers only until a global pooling operation has occurred (e.g. GlobalMaxPooling2D). Then Dense layers etc. can be used because the size is now fixed.
  3. Use a batch size of 1 only. This avoids dealing with mixed sizes within a batch.
  4. Write a small custom Sequence that creates batches of size 1 from the list of inputs. I did this to avoid dealing with different sizes inside a single Numpy array.
  5. Use Model.fit_generator on your custom Sequence for training and validation. (v.s. Model.fit)
  6. For some reason, Model.predict_generator popped even when using the Sequence as above. I had to resort to using Model.predict on individual inputs.

Note that calls to Model.predict did complain about performance - which is unsurprising given the inefficiency of the solution - but it works!

$\endgroup$
3
  • 1
    $\begingroup$ Tensorflow 2.4 and now dropped the usage of model.fit_generator() and model.predict_generator(). Instead you can directly use model.fit() and model.predict() $\endgroup$
    – Swaroop
    Apr 15, 2021 at 14:42
  • 3
    $\begingroup$ The biggest downside of using inputs with mixed sizes is to restrict with a batch size of 1. Does anyone have a workaround to this problem? $\endgroup$
    – Swaroop
    Apr 15, 2021 at 14:45
  • $\begingroup$ Sadly, I never did find one. Anybody else? $\endgroup$
    – J Trana
    Apr 16, 2021 at 4:32
-2
$\begingroup$

Yes, simply select an appropriate backbone network which doesn't rely on the size of the input image to be some precise value -- most networks satisfy this criteria.

$\endgroup$
1
  • 10
    $\begingroup$ You are not wrong, but your answer is not very informative - what about expanding it to explain why most modern CNN can work with variable size images? Also, what are the limits to this variability (for example, don't try to mix different size images in the same mini-batch...)? Most people coming from either old-fashioned MLPs (input length is fixed) or old-fashioned CNNs (AlexNet & VGG-1X), with their pesky Flatten layers, don't understand how the modern CNNs can in principle take images of any size. $\endgroup$
    – DeltaIV
    Jan 24, 2019 at 7:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.