5
$\begingroup$

Convolutional networks have been proven to work very well detecting a shape independently of where it is in the image, which is referred as translational invariance.

In the case where the position of an object in an image contains information for a classification problem, are convolutional networks still a good method? For example, if we have images of this kind:

Cat habits

The position of the cat in the images is related with the time of the day, and because of that, we can infer if the cat is having lunch or dinner.

Is convolution still a good candidate to solve this problem or are there more appropriate methods? In case convolution fits this purpose, might a fully connected layer be a better approach to solve the spatial variability?

Edit: This is an oversimplification of the problem where images contain entangled and complex patterns which cannot be easily isolated.

$\endgroup$
2
  • 1
    $\begingroup$ Once you detect a shape, it's often easy to locate it. $\endgroup$
    – whuber
    Commented Apr 13, 2016 at 22:02
  • $\begingroup$ I have a related question: stats.stackexchange.com/questions/287580/… I think you just have to not pool and have a stride of 1, but then I am not sure if this means you still have a CNN. $\endgroup$
    – ashley
    Commented Jun 27, 2017 at 14:22

2 Answers 2

10
+100
$\begingroup$

Yes, convolutional networks are still a good candidate. As you know, the way they work is detecting more and more complex features, starting from edges and colors and detecting textures and other concepts in deeper layers. To detect a feature (let's say a cat head), it is not enough to know that relevant lower level features are present in the region, but their relative position also plays a role (e.g. the eyes should be lower than ears). Convolutional networks do capture this information, and if the relative position of an object plays a role for classification, they should discover it.

As you already mentioned, fully-connected layers encode this positional information even better, and they are used after convolution layers in networks for classification. Possibly even better would be using a "locally-connected layer", which is a combination of a conv-layer and a fully-connected one: it performs a convolution operation, but at every position it learns a separate convolution kernel.

Finally, if you know that position of some objects is crucial for the classification, consider splitting the problem in two parts. Firstly, train a network that can detect these relevant objects (for example a fully convolutional network with low output resolution), and secondly add an extra layer on top of this (locally connected for example) to do the classification. It has been demonstrated by (Gulcehre and Bengio, 2013) that such subproblem splitting can make the difference between easy and impossible: Knowledge Matters: Importance of Prior Information for Optimization.


Finally, I would correct that the phenomenon you are referring to is not called translation invariance but equivariance. A great explanation of the difference is here:

Invariance to a transformation means if you take the input and transform it then the representation you get is the same as the representation of the original, i.e. represent(x) = represent(transform(x))

Equivariance to a transformation means if you take the input and transform it then the representation you get is a transformation of the representation of the original, i.e. transform(represent(x)) = represent(transform(x)).

$\endgroup$
2
  • $\begingroup$ Actually the classification (i.e., the CNN output) is (approximately) translation invariant ( not just equivariant) in a lot of CNNs (for example in ResNet-50 trained on ImageNet). In other words, the classification remains “black cat” even if the position of the cat in the picture is changed by some amount of translation. $\endgroup$
    – DeltaIV
    Commented Feb 1, 2018 at 14:37
  • 1
    $\begingroup$ @DeltaIV You are right to point this out. That's due to the fact that the output is of the shape 1*1*num_classes, therefore transform(represent(x)) does not really make any sense. Also, they often use fully connected layers or some pooling as the final classifier, which is responsible for the invariance. However, the individual conv. layers are equivariant to translations. $\endgroup$ Commented Feb 1, 2018 at 14:44
0
$\begingroup$

I guess the simplest most intuitive way is to divide the original image to four sub-images then pass those images to the conv-net and see which one contains the image of the cat. And then keep applying the same method to each one of the sub-images depending on the precision of the position needed. Another simple way would be to put in place another "module" (not necessarily a conv-net based one) that detects where a known shape is in the image. That way, once you've detected that a cat is present in the picture (with your conv-net), you'd call that module in order to get where the closest shape to a cat figure is in the image.

$\endgroup$
1
  • $\begingroup$ Thanks for your response, but as I explained in my question, the example presented a simplification used to present the problem. I'm interested in a generic solution which works without an a-priori knowledge of the image structure. $\endgroup$
    – prl900
    Commented Feb 1, 2018 at 7:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.