Convolutional Neural Network Scale Sensitivity

Question

For the sake of example, lets suppose we're building an age estimator, based on the picture of a person. Below we have two people in suits, but the first one is clearly younger than the second one.

_{(source: tinytux.com)}

There are plenty of features that imply this, for example the face structure. However the most telling feature is the ratio of head size to body size:

_{(source: wikimedia.org)}

So suppose we've trained a CNN regression to predict the age of the person. In a lot of the age predictors that I've tried, the above image of the kid seems to fool the predictions into thinking he's older, because of the suit and likely because they rely primarily the face:

I'm wondering just how well can a vanilla CNN architecture infer the ratio of head to torso?

Compared to a regional RCNN, which is able to get bounding boxes on the body and head, will the vanilla CNN always perform worse?

Just before global flattening in the vanilla CNN (i.e. just after all convolutions), each output has a corresponding receptive field, which should have a sense of scale. I know that faster RCNN exploits this by making bounding box proposals exactly at this stage, so that all prior convolutional filters automatically train to all scales.

So, I would think that the vanilla CNN should be able to infer the ratio of head to torso size? Is this right? If so, is the only benefit of using a faster RCNN framework to exploit the fact that may have been pre-trained on detecting people?

Answer 1

CNNs are too large a class of models, to answer this question. LeNet, AlexNet, ZFNet and VGG16 will behave very differently than GoogLeNet, which was built specifically to do most of what R-CNN do, with a CNN architecture (you may know GoogLeNet with the name of Inception, even though strictly speaking Inception is just the basic unit (subnetwork) upon which GoogLeNet is built). Finally, ResNets will behave differently. And all these architectures were not built to classify age classes, but the 1000 ImageNet classes, which don't contain age classes for humans. One could use transfer learning (if you have enough training images) to train one of the widely available trained models above, and see how they perform. In general, however, especially the older architectures (let's say up to VGG16) have an hard time learning "global features" which require to learn about "head" (already a complex feature), "torso" (another complex feature) and their ratio (which also requires that the two features are in a certain spatial relationship). This kind of stuff is what Capsule Networks should have been able to do.

Convnets were born to do exactly the opposite: be sensitive to local features, and relatively insensitive to their relative position/scale. A good Convnet should recognize "white cat" whether the picture is a close-up or an American shot. Combining convolutional layers (which are sensitive to local features) with pooling layers (which remove part of the sensitivity to variation in scale or translation of the image) gives you an architecture which in its most basic form is not great at learning the kind of spatial relationships among objects which you're looking for. There was an example somewhere (but I can't find it anymore) where, after splitting a cat image in various rectangular nonoverlapping tiles and putting them together in a random order, the CNN would keep identifying the image as cat. This indicates that CNNs are more sensitive to local features (textures or something like that) than to the spatial relationship among high level features. See also the Capsule networks paper for some discussion of this. Hinton also showed an example of this in a video about the limits of convnets.

My wild guess is that one of the recent architectures would be perfectly capable (given enough data) of discerning men from children, but not because of a "threshold" on a metric relationship among high level features such as "head" and "torso". It would learn some statistical regularity, maybe completely unnoticeable to humans, which separates adult images from child images in the training set.

Answer 2

Firstly, thanks for posting a very interesting question.

To answer it shortly, a vanilla convnet trained end-2-end to predict age from a photo will be generally prone to mis-classify images such as the one you posted. Secondly, note that accurately estimating the age of a person is a nearly impossible task¹.

The main difference from your proposed approach using some object detectors (be it RCNN, Faster RCNN, YOLO or SSD) is that you are using different information to train the models. The CNN is trained only on images and needs to find out all the necessary features itself. It is most likely going to find various facial features, but it will also rely on clothing and perhaps scene features (kids may be often in the picture with some toys, adults will be more likely in office environments, etc.). These features will not be robust to your counterexample.

On the other hand, if you train the network to explicitly detect objects as "torso" and "head", you are providing extra information that these objects are important for the task, and thus simplify the problem².

While the approach of detecting head and torso and then evaluation the size ratio of the bounding boxes sounds interesting, I can see several obstacles:

1. Obtaining data: I am not aware of the availability of large dataset where both age and bounding boxes would be present.
2. Imperfect FOV: In most images (e.g. both your examples), the people are not displayed whole. You would have to deal with the fact that the torso bounding boxes would not be always perfect simply because part of the person is not in the image and the net would have to guess how large part is missing (and the ground truth bounding boxes would most likely not capture this information). Also, the aforementioned object detectors don't always handle predictions of partial objects properly. This might introduce too much noise in the model.
3. Various poses: The torso-to-head ratio would be very different for people viewed frontally and from the side.
4. Adults: It seems the ratio works well to predict ages between 0-21, but I don't see how it would help to predict ages of adults (I suppose the ratio does not change in higher age).

All these problems suggest that the head-to-torso ratio approach is also not going to work perfectly, although it might be more robust to your particular counterexample.

I guess the best way to perform this task would be to 1) detect the face, 2) predict age only from the facial crop (removes potentially misleading information). Note that some R-CNN-like architecture using ROI-pooling could be trained to do this end-2-end.

---

¹ Even using very sophisticated medical methods (which are arguably much more informative than a photo of the person) this is not possible to do accurately. See this Quora thread for more information.

² Check the article Knowledge Matters: Importance of Prior Information for Optimization for an example how providing some intermediate knowledge about the task can greatly simplify learning.

Answer 3

Well, it all depends on how your dataset is constructed. From my experience neural networks tend to go for simplest explanations. And inferring the age from the outfit is actually simpler than using head to body ratio. If you can expand your dataset having this in mind your CNN should work as expected.