2
$\begingroup$

I'm looking for some references on the following problem. You're given a pre-trained classifier network, say ResNet-50, on images that are 255x255, from which you can extract the last fully connected layer (2048 dimensional) to get features. The goal is to then leverage the pre-trained model for a different classification task on large images, say 640x480.

The obvious thing to do is to split the large image into $N$ pieces (for example $N=4$ quadrants), each of which gets fed into the original model, which gives $N$ number of outputs of size 2048 each. Then you slap on a few additional fully connected layers to perform your classification task. I'm assuming here that the intelligent thing to do is to share weights between the $N$ outputs, to reduce the computational complexity and treat each piece equally.

This has a disadvantage in that you are artificially splitting the image into pieces, and ending up with a very large embedding (even with the above weight sharing scheme).

The alternative would be to use a pre-trained bounding box model (faster-RCNN, etc.) , from which you can extract proposal regions, and then feed each proposal region into a common object classifier. This has the advantage of no artificial image splitting, but is disadvantageous due to the sheer number of proposals.

Are the above two schemes essentially the only options? I'd really appreciate some references!

Improve this question
$\endgroup$

3 Answers 3

Reset to default
3
$\begingroup$

None of the operations (convolutions and pooling) in ResNet depend on the actual size of the image or feature maps, so there is nothing stopping you from just feeding a different sized image in and letting the global averaging layer before the fully-connected layers take care of the rest.

This allows the full information from the higher resolution image to be utilized. The only disadvantage is that you'll need a lot of memory when dealing with very large images, but I don't think 640x480 will be a problem. Some fine-tuning will be advisable of course.

Improve this answer
$\endgroup$
3
$\begingroup$

I would simply re-size the image and use the same neural network..see the example in

https://keras.io/applications/#usage-examples-for-image-classification-models

with elephant.jpg. It seems like your problem is actually object detection and not simply classification (a classification task typically means one object/ image). If it is object detection "Classification" in its naive sense does not really have a physical meaning. What does the classification probability of an image as being there mean if there are multiple objects present.

This implies that you probably should be using a ROI method in the first place, and naive Resnet50 may not be meaningful in the first place. You can use region proposal based methods (note that even in region proposal based methods, you do in fact re-scale the region proposals to meet the input dimensions for the input image).

Your intuitive idea of splitting the image into quadrants is actually on the right track :). It is a special case of YOLO (slide 83 of http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture11.pdf). You dont actually split it into 4 quadrants, but consider overlapping chunkgs of size 255*255*3 which move over the image

Improve this answer
$\endgroup$
2
  • $\begingroup$ Resizing the image causes a massive drop in resolution and doesn't work well when there are small objects in the original image. That's why I'm interested in whether it makes more sense to apply a region-based model to take care of scale issues. $\endgroup$
    – Alex R.
    Feb 1, 2018 at 18:20
  • $\begingroup$ Ahh, so your question is not actually classification, it is more precisely object detection. I have edited my post $\endgroup$
    – Sid
    Feb 1, 2018 at 23:42
0
$\begingroup$

You can consider using Autoencoders to convert 640x480 images to 255x255 features and then feed them directly to the pre-trained networks.

This is better than using faster RCNN & other regions proposal networks.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Autoencoding would work if resolution isn’t that important. But the issue here is that you’d also need to autoencode while preserving the form of inputs that the smaller network expects. So this may be reasonable in simpler cases but seems difficult when resolution and scale becomes important. $\endgroup$
    – Alex R.
    Feb 1, 2018 at 6:30
  • $\begingroup$ If you're looking to solve an object detection problem, resolution matters more when compared to classification problems. Is there any reason why you're not willing to resize 640x480 directly to 255x255? Have you tried this already? $\endgroup$
    – Avis
    Feb 1, 2018 at 6:36
  • $\begingroup$ I know for a fact that if I resize the image, finer details will be lost. This is an issue of scale. As an example, imagine there's a small car in the bottom right of the original image. Resizing it would make it too tiny to detect properly. $\endgroup$
    – Alex R.
    Feb 1, 2018 at 18:20
  • $\begingroup$ I can understand your concern for resizing directly. As @shimao has mentioned you can try to fit in Fully convolutional networks so that they're not dependent on the input image resolution & fine tune few final layers. It will be better than using Autoencoders. $\endgroup$
    – Avis
    Feb 1, 2018 at 23:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.