I saw a term describing the feature detectors, i.e. shift invariant. What is that mean?
Paper: 1989 Generalization and Network Design Strategies
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this communityI saw a term describing the feature detectors, i.e. shift invariant. What is that mean?
Paper: 1989 Generalization and Network Design Strategies
For CNNs, I think it means the invariance to small* displacements of the input image. For example in the character recognition task, if you train the system by shifting (i.e. sliding the images to left/right and up/down) a little bit, you learn a more generalizable detector, that works under difficult conditions, i.e. when the character is not perfectly aligned to the center of the image. Similar precautions are also taken for rotation, scale, etc.
$^*$ I Googled to be sure about "small" and saw a similar discussion here, it made me realize that CNNs can be resistant to big displacements too, since the pooling process summarizes the local features in a smaller vector (that is representing the whole), it doesn't matter where you see the objects in the image.
Shift-invariance: this means that if we shift the input in time (or shift the entries in a vector) then the output is shifted by the same amount
http://pillowlab.princeton.edu/teaching/mathtools16/slides/lec22_LSIsystems.pdf
Adding to the previous answers, in math terms:
$$f(x)\implies Transform\ T\implies g(x)$$
If the transform is shift invariant, then:
$$f(x+a)\implies Transform\ T\implies g(x+a)$$
I.e. the $a$ shift is carried through the transform "untouched"
Source: This excellent ETH lecture by Prof. Buhmann
Shift-Invariance arises from Computer Vision tasks such as Image Classification.
At a high level it means that the classifier should not be affected by the position of the object (e.g. cat) in the image. [From Bronstein et. al. Geometric DL]
A little more: MLPs do not have this property. The claim that CNNs are shift-invariant is contested by Bronstein et. al., CNNs are shift-equivariant ("a shift of the input to a convolutional layer produces a shift in the output feature maps by the same amount"). What is shift invariant in traditional CV architectures are the pooling layers.