# Convolutional Neural Network Scale Sensitivity

For the sake of example, lets suppose we're building an age estimator, based on the picture of a person. Below we have two people in suits, but the first one is clearly younger than the second one.

(source: tinytux.com)

There are plenty of features that imply this, for example the face structure. However the most telling feature is the ratio of head size to body size:

(source: wikimedia.org)

So suppose we've trained a CNN regression to predict the age of the person. In a lot of the age predictors that I've tried, the above image of the kid seems to fool the predictions into thinking he's older, because of the suit and likely because they rely primarily the face:

I'm wondering just how well can a vanilla CNN architecture infer the ratio of head to torso?

Compared to a regional RCNN, which is able to get bounding boxes on the body and head, will the vanilla CNN always perform worse?

Just before global flattening in the vanilla CNN (i.e. just after all convolutions), each output has a corresponding receptive field, which should have a sense of scale. I know that faster RCNN exploits this by making bounding box proposals exactly at this stage, so that all prior convolutional filters automatically train to all scales.

So, I would think that the vanilla CNN should be able to infer the ratio of head to torso size? Is this right? If so, is the only benefit of using a faster RCNN framework to exploit the fact that may have been pre-trained on detecting people?

• Do you know where exactly your age recongnizer fails? Why do you think it's head size proportion? Did you look at the output of middle layers? – Aksakal Feb 20 '18 at 0:51
• @Aksakal I don't think he experimented training a CNN. From what I've understood, he's been making tests with existing web services: " In a lot of the age predictors that I've tried[..]". – DeltaIV Feb 20 '18 at 11:08

Firstly, thanks for posting a very interesting question.

To answer it shortly, a vanilla convnet trained end-2-end to predict age from a photo will be generally prone to mis-classify images such as the one you posted. Secondly, note that accurately estimating the age of a person is a nearly impossible task1.

The main difference from your proposed approach using some object detectors (be it RCNN, Faster RCNN, YOLO or SSD) is that you are using different information to train the models. The CNN is trained only on images and needs to find out all the necessary features itself. It is most likely going to find various facial features, but it will also rely on clothing and perhaps scene features (kids may be often in the picture with some toys, adults will be more likely in office environments, etc.). These features will not be robust to your counterexample.

On the other hand, if you train the network to explicitly detect objects as "torso" and "head", you are providing extra information that these objects are important for the task, and thus simplify the problem2.

While the approach of detecting head and torso and then evaluation the size ratio of the bounding boxes sounds interesting, I can see several obstacles:

1. Obtaining data: I am not aware of the availability of large dataset where both age and bounding boxes would be present.
2. Imperfect FOV: In most images (e.g. both your examples), the people are not displayed whole. You would have to deal with the fact that the torso bounding boxes would not be always perfect simply because part of the person is not in the image and the net would have to guess how large part is missing (and the ground truth bounding boxes would most likely not capture this information). Also, the aforementioned object detectors don't always handle predictions of partial objects properly. This might introduce too much noise in the model.
3. Various poses: The torso-to-head ratio would be very different for people viewed frontally and from the side.
4. Adults: It seems the ratio works well to predict ages between 0-21, but I don't see how it would help to predict ages of adults (I suppose the ratio does not change in higher age).

All these problems suggest that the head-to-torso ratio approach is also not going to work perfectly, although it might be more robust to your particular counterexample.

I guess the best way to perform this task would be to 1) detect the face, 2) predict age only from the facial crop (removes potentially misleading information). Note that some R-CNN-like architecture using ROI-pooling could be trained to do this end-2-end.

1 Even using very sophisticated medical methods (which are arguably much more informative than a photo of the person) this is not possible to do accurately. See this Quora thread for more information.

2 Check the article Knowledge Matters: Importance of Prior Information for Optimization for an example how providing some intermediate knowledge about the task can greatly simplify learning.

CNNs are too large a class of models, to answer this question. LeNet, AlexNet, ZFNet and VGG16 will behave very differently than GoogLeNet, which was built specifically to do most of what R-CNN do, with a CNN architecture (you may know GoogLeNet with the name of Inception, even though strictly speaking Inception is just the basic unit (subnetwork) upon which GoogLeNet is built). Finally, ResNets will behave differently. And all these architectures were not built to classify age classes, but the 1000 ImageNet classes, which don't contain age classes for humans. One could use transfer learning (if you have enough training images) to train one of the widely available trained models above, and see how they perform. In general, however, especially the older architectures (let's say up to VGG16) have an hard time learning "global features" which require to learn about "head" (already a complex feature), "torso" (another complex feature) and their ratio (which also requires that the two features are in a certain spatial relationship). This kind of stuff is what Capsule Networks should have been able to do.

Convnets were born to do exactly the opposite: be sensitive to local features, and relatively insensitive to their relative position/scale. A good Convnet should recognize "white cat" whether the picture is a close-up or an American shot. Combining convolutional layers (which are sensitive to local features) with pooling layers (which remove part of the sensitivity to variation in scale or translation of the image) gives you an architecture which in its most basic form is not great at learning the kind of spatial relationships among objects which you're looking for. There was an example somewhere (but I can't find it anymore) where, after splitting a cat image in various rectangular nonoverlapping tiles and putting them together in a random order, the CNN would keep identifying the image as cat. This indicates that CNNs are more sensitive to local features (textures or something like that) than to the spatial relationship among high level features. See also the Capsule networks paper for some discussion of this. Hinton also showed an example of this in a video about the limits of convnets.

My wild guess is that one of the recent architectures would be perfectly capable (given enough data) of discerning men from children, but not because of a "threshold" on a metric relationship among high level features such as "head" and "torso". It would learn some statistical regularity, maybe completely unnoticeable to humans, which separates adult images from child images in the training set.

• I appreciate your answer, but, I'm having trouble agreeing. RCNN architectures have essentially the same structure of filters as object convnets, for example VGG and Resnet. And since RCNN can detect scale and relative position, it follows that VGG and Resnet should also be able to detect scale. However, RCNN architectures rely on box proposals, of which they make thousands per image, after which each box proposal is evaluated. So it seems like if I incorporate at least some of these box proposals, a vanilla CNN should detect scale better. I'm just not sure if it's necessary to do so. – Alex R. Feb 12 '18 at 18:15
• RCNN are not CNN. Not only you miss the selective search for the bounding boxes, but you also miss the linear SVM and the bounding box regressor stages. Also, there is a big difference between the ability to detect scale of AlexNet (which is the CNN used in the original RCNN paper), or VGG, and the ability of GoogLeNet or ResNet: GoogLeNet was developed precisely to do what RCNN do. I think both GoogLeNet and ResNet would be able to classify age, but there's no way to know if they would do it by using a feature which makes sense to us (head to torso ratio) or by finding some statistical 1/ – DeltaIV Feb 12 '18 at 20:07
• 2/ regularities which a human would never notice. I'd suggest you to experiment and try, but unfortunately only building the image database would be a research project in itself (unless you work in a fashion company). – DeltaIV Feb 12 '18 at 20:08
• My apologies for the confusion. I know there are like, 20 different RCNN architectures out there, each claiming the other ones are obsolete. – Alex R. Feb 12 '18 at 21:05
• blog.piekniewski.info/2016/12/29/can-a-deep-net-see-a-cat Also, a different issue, but still related to texture matching, and from more respected researchers, arxiv.org/pdf/1703.06857 – DeltaIV Feb 17 '18 at 8:06

Well, it all depends on how your dataset is constructed. From my experience neural networks tend to go for simplest explanations. And inferring the age from the outfit is actually simpler than using head to body ratio. If you can expand your dataset having this in mind your CNN should work as expected.