I don't have computer vision background, yet when I read some image processing and convolutional neural networks related articles and papers, I constantly face the term, translation invariance, or translation invariant.
Or I read alot that the convolution operation provides translation invariance?!! what does this mean?
I myself always translated it to myself as if it means if we change an image in any shape, the actual concept of the image doesn't change.
For example if I rotate an image of a lets say tree, it's again a tree no matter what I do to that picture.
And I myself consider all operations that can happen to an image and transform it in a way (crop it, resize it, gray-scale it,color it etc...) to be this way. I have no idea if this is true so I would be grateful if anyone could explain this to me .
5 Answers
You're on the right track.
Invariance means that you can recognize an object as an object, even when its appearance varies in some way. This is generally a good thing, because it preserves the object's identity, category, (etc) across changes in the specifics of the visual input, like relative positions of the viewer/camera and the object.
The image below contains many views of the same statue. You (and well-trained neural networks) can recognize that the same object appears in every picture, even though the actual pixel values are quite different.
Note that translation here has a specific meaning in vision, borrowed from geometry. It does not refer to any type of conversion, unlike say, a translation from French to English or between file formats. Instead, it means that each point/pixel in the image has been moved the same amount in the same direction. Alternately, you can think of the origin as having been shifted an equal amount in the opposite direction. For example, we can generate the 2nd and 3rd images in the first row from the first by moving each pixel 50 or 100 pixels to the right.
One can show that the convolution operator commutes with respect to translation. If you convolve $f$ with $g$, it doesn't matter if you translate the convolved output $f*g$, or if you translate $f$ or $g$ first, then convolve them. Wikipedia has a bit more.
One approach to translation-invariant object recognition is to take a "template" of the object and convolve it with every possible location of the object in the image. If you get a large response at a location, it suggests that an object resembling the template is located at that location. This approach is often called template-matching.
Invariance vs. Equivariance
Santanu_Pattanayak's answer (here) points out that there is a difference between translation invariance and translation equivariance. Translation invariance means that the system produces exactly the same response, regardless of how its input is shifted. For example, a face-detector might report "FACE FOUND" for all three images in the top row. Equivariance means that the system works equally well across positions, but its response shifts with the position of the target. For example, a heat map of "face-iness" would have similar bumps at the left, center, and right when it processes the first row of images.
This is is sometimes an important distinction, but many people call both phenomena "invariance", especially since it is usually trivial to convert an equivariant response into an invariant one--just disregard all the position information).
-
2$\begingroup$ Could you clarify how translation invariance is achieved with CNN? The activations of a convolutional layer in a CNN are not invariant under translations: they move around as the image moves around (i.e., they are equivariant, rather than invarianct, to translations). Those activations are usually fed into a pooling layer, which also isn't invariant to translations. And pooling layer may feed into a fully connected layer. Do the weights in a fully connected layer somehow change transalation equivariant to translation invariant behavior? $\endgroup$– maxOct 16, 2017 at 8:39
-
1$\begingroup$ @max, Pooling does increase translation invariance, especially max-pooling(!), which completely disregards spatial information within the pooling neighborhood. See Chapter 9 of Deep Learning deeplearningbook.org/contents/convnets.html (starting on p. 335). This idea is also popular in neuroscience--the HMAX model (e.g., here: maxlab.neuro.georgetown.edu/docs/publications/nn99.pdf) uses a combination of averaging and max-pooling to generate translation (and other kinds of ) invariance. $\endgroup$ Oct 16, 2017 at 17:16
-
1$\begingroup$ Oh right, pooling provides invariance over small translations (I was thinking about larger shifts, but perhaps each successive layer of pooling can handle progressively larger shifts). But what about the fully convolutional networks? Without pooling, what provides (at least approximate) invariance? $\endgroup$– maxOct 16, 2017 at 23:10
-
$\begingroup$ How do one explain that "speech is translational invariant only along time-axis, but not frequency axis", what does it mean? $\endgroup$– GeneNov 14, 2017 at 15:04
-
1$\begingroup$ @Fredom, that might be better as a new question, but in brief--the audio signal sounds the same even when you shift it forwards in time (e.g., by adding a bunch of silence at the beginning). However, if you shift it in the frequency domain, it sounds different: not only is the spectrum shifted, but relationships between frequencies (e.g., harmonics) are also distorted. $\endgroup$ Nov 14, 2017 at 17:24
I think there is some confusion about what is meant by translational invariance. Convolution provides translation equivariance meaning if an object in an image is at area A and through convolution a feature is detected at the output at area B, then the same feature would be detected when the object in the image is translated to A'. The position of the output feature would also be translated to a new area B' based on the filter kernel size. This is called translational equivariance and not translational invariance.
@Santanu
While your answer is correct in part and leads to confusion. It is true that Convolutional layers themselves or output feature maps are translation equivariant. What the max-pooling layers do is provide some translation invariance as @Matt points out.
That is to say, the equivariance in the feature maps combined with max-pooling layer function leads to translation invariance in the output layer (softmax) of the network. The first set of images above would still produce a prediction called "statue" even though it has been translated to the left or right. The fact that the prediction remains "statue" (i.e. the same) despite translating the input means the network has achieved some translation invariance.
-
$\begingroup$ I'm not so sure that pooling leads to translation invariance. $\endgroup$– AksakalNov 15, 2017 at 16:20
-
1$\begingroup$ Moderately it does. Remember the max pooling operator takes the maximum pixel value as its output in a given window. This mathematically necessitates some invariance since the spatial location of the maximum pixel values is irrelevant (within some tolerance). $\endgroup$– mr eNov 15, 2017 at 16:26
-
$\begingroup$ And why translation invariance is useful in a CNN? If we care only about detecting features, we can just remove all max pool layers and use a global max pool layer after the final convolutional layer. Why bothering with max pooling in intermediate layers? $\endgroup$– ado sarJul 15, 2023 at 0:09
The answer is actually trickier than it appears at first. Generally, the translational invariance means that you would recognize the object regerdless of where it appears on the frame.
In the next picture in frame A and B you would recognize the word "stressed" if your vision supports translation invariance of words.
I highlighted the term words because if your invariance is only supported on letters, then the frame C will also be equal to frames A and B: it has exactly the same letters.
In practical terms, if you trained your CNN on letters, then things like MAX POOL will help to achieve the translation invariance on letters, but may not necessarily lead to translation invariance on words. Pooling pulls out the feature (that's extracted by a corresponding layer) without relation to the location of other features, so it'll lose the knowledge of relative position of letters D and T and words STRESSED and DESSERTS will look the same.
The term itself is probably from physics, where translational symmetry means that the equations stay the same regardless of translation in space.
-
$\begingroup$ Couldn't we achieve translation invariance with global pooling after the final convolutional layer? Why bothering with max pool in intermediate layers? $\endgroup$– ado sarJul 15, 2023 at 0:11
I will answer you briefly, translation invariant means that the algorithm will recognize the object even if you shifted its position from any place in the picture to any other place, lets say , we have a cat in photo with grass background , if you shifted the cat to the corner of the image or any other place , model still recognize , it is not affected by the translation ( shifting )
