I saw a term describing the feature detectors, i.e. shift invariant. What is that mean?
Paper: 1989 Generalization and Network Design Strategies
$\begingroup$
$\endgroup$
4
-
$\begingroup$ Do you see what shift invariance means in general? It is a property of an algorithm. $\endgroup$– user603Oct 28, 2014 at 11:20
-
$\begingroup$ so what is this property? $\endgroup$– RockTheStarOct 28, 2014 at 17:03
-
2$\begingroup$ That the result of the algorithm is not changed is you shift the inputs. For example, considering the standard deviation, since for any vector $\pmb x=\{x_1,\ldots,x_n\}$ and scalar $\delta$, it holds that $\mbox{sd}(\pmb x+\delta)=\mbox{sd}(\pmb x)$ so we say that the standard deviation is shift invariant. The mean, for example, is not shift invariant. $\endgroup$– user603Oct 28, 2014 at 17:18
-
$\begingroup$ Duda, Hart, & Stork, "Pattern Classification," p. 317 says that deep networks can more easily learn shift invariance because if, e.g., one layer can handle 2 pixels of shift, then two layers could learn shift invariance of four pixels. Is anyone aware of a longer exploration of this topic? $\endgroup$– Josiah YoderMay 17, 2018 at 14:49
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
0
For CNNs, I think it means the invariance to small* displacements of the input image. For example in the character recognition task, if you train the system by shifting (i.e. sliding the images to left/right and up/down) a little bit, you learn a more generalizable detector, that works under difficult conditions, i.e. when the character is not perfectly aligned to the center of the image. Similar precautions are also taken for rotation, scale, etc.
$^*$ I Googled to be sure about "small" and saw a similar discussion here, it made me realize that CNNs can be resistant to big displacements too, since the pooling process summarizes the local features in a smaller vector (that is representing the whole), it doesn't matter where you see the objects in the image.
$\begingroup$
$\endgroup$
1
Shift-invariance: this means that if we shift the input in time (or shift the entries in a vector) then the output is shifted by the same amount
http://pillowlab.princeton.edu/teaching/mathtools16/slides/lec22_LSIsystems.pdf
-
2$\begingroup$ An image has no "time" domain / axis ... $\endgroup$ Nov 12, 2021 at 10:04
$\begingroup$
$\endgroup$
Adding to the previous answers, in math terms:
$$f(x)\implies Transform\ T\implies g(x)$$
If the transform is shift invariant, then:
$$f(x+a)\implies Transform\ T\implies g(x+a)$$
I.e. the $a$ shift is carried through the transform "untouched"
Source: This excellent ETH lecture by Prof. Buhmann
$\begingroup$
$\endgroup$
Shift-Invariance arises from Computer Vision tasks such as Image Classification.
At a high level it means that the classifier should not be affected by the position of the object (e.g. cat) in the image. [From Bronstein et. al. Geometric DL]
A little more: MLPs do not have this property. The claim that CNNs are shift-invariant is contested by Bronstein et. al., CNNs are shift-equivariant ("a shift of the input to a convolutional layer produces a shift in the output feature maps by the same amount"). What is shift invariant in traditional CV architectures are the pooling layers.
