The convolutional layers and pooling layers themselves are independent of the input dimensions. However, the output of the convolutional layers will have different spatial sizes for differently sized images, and this will cause an issue if we have a fully connected layer afterwards (since our fully connected layer requires a fixed size input). There are several solutions to this:
1. Global Pooling: Avoid fully connected layers at the end of the convolutional layers, and instead use pooling (such as Global Average Pooling) to reduce your feature maps from a shape of (N,H,W,C) (before global pool) to shape (N,1,1,C) (after global pool), where:
N = Number of minibatch samples
H = Spatial height of feature map
W = Spatial width of feature map
C = Number of feature maps (channels)
As can be seen, the output dimensionality (N*C) is now independent of the spatial size (H,W) of the feature maps. In case of classification, you can then proceed to use a fully connected layer on top to get the logits for your classes.
2. Variable sized pooling: Use variable sized pooling regions to get the same feature map size for different input sizes.
3. Crop/Resize/Pad input images: You can try to rescale/crop/pad your input images to all have the same shape.
In the context of transfer learning, you might want to use differently sized inputs than the original inputs that the model was trained with. Here are some options for doing so:
4. Create new fully connected layers:
You can ditch the original fully connected layers completely and initialize a new fully connected layer with the dimensionality that you need, and train it from scratch.
5. Treat the fully connected layer as a convolution:
Normally, we reshape the feature maps from (N,H,W,C) to (N,H*W*C) before feeding it to the fully connected layer. But you can also treat the fully connected layer as a convolution with a receptive field of (H,W). Then, you can just convolve this kernel with your feature maps regardless of their size (use zero padding if needed) [http://cs231n.github.io/transfer-learning/