About CNN, kernels and scale/rotation invariance

Question

I have a couple of questions that are confusing me regarding the CNN.
1) The features extracted using CNN are scale and rotation invariant?
2) The Kernels we use to convolution with our data are already defined in the literature? what kind of these kernels are? is it different for every application?

Answer 1

1) The features extracted using CNN are scale and rotation invariant?

A feature in itself in a CNN is not scale or rotation invariant. For more details, see: Deep Learning. Ian Goodfellow and Yoshua Bengio and Aaron Courville. 2016: http://egrcc.github.io/docs/dl/deeplearningbook-convnets.pdf ; http://www.deeplearningbook.org/contents/convnets.html:

Convolution is not naturally equivariant to some other transformations, such as changes in the scale or rotation of an image. Other mechanisms are necessary for handling these kinds of transformations.

It is the max pooling layer that introduces such invariants:

2) The Kernels we use to convolution with our data are already defined in the literature? what kind of these kernels are? is it different for every application?

The kernels are learnt during the training phase of the ANN.

Answer 2

I think there are a couple of things confusing you, so first things first.

Given a signal $x[n]$, and a kernel (also called a filter) $h[n]$, then the convolution of $x[n]$ with $h[n]$ is written as $y[n] = (x \star h)[n]$, and is computed via a sliding dot-product, mathematically given by:

$$ y[n] = \sum_{m=-\infty}^{\infty}x[m] \ h[n-m] $$

The above if for one-dimensional signals, but the same can be said for images, which are just two-dimensional signals. In that case, the equation becomes:

$$ I_{new}[r,c] = \sum_{u=-\infty}^{\infty}\sum_{v=-\infty}^{\infty} I_{old}[u,v] \ k[r-u, c-v] $$

Pictorially, this is what is happening:

At any rate, the thing to keep in mind, is that the kernel, in actually learned during training a Deep Neural Network (DNN). A kernel is just going to be what you convolve your input with. The DNN will learn the kernel, such that it brings out certain facets of the image (or previous image), that are going to be good for lowering the loss of your target objective.

This is the first crucial point to understand: Traditionally people have designed kernels, but in Deep Learning, we let the network decide what the best kernel should be. The one thing we do specify however, is the kernel dimensions. (This is called a hyperparameter, for example, 5x5, or 3x3, etc).

Answer 3

Many authors including Geoffrey Hinton (who proposes Capsule net) try to solve the issue, but qualitatively. We try to address this issue quantitatively. By having all convolution kernels be symmetric (dihedral symmetry of order 8 [Dih4] or 90-degree increment rotation symmetric, et al) in the CNN, we would provide a platform for the input vector and resultant vector on each convolution hidden layer be rotated synchronously with the same symmetric property (i.e., Dih4 or 90-increment rotation symmetric, et al). Additionally, by having the same symmetric property for each filter (i.e., fully connected but weighs sharing with the same symmetric pattern) on the first flatten layer, the resultant value on each node would be quantitatively identical and lead to the CNN output vector the same as well. I called it transformation-identical CNN (or TI-CNN-1). There are other methods that can also construct transformation-identical CNN using symmetric input or operations inside the CNN (TI-CNN-2). Based on the TI-CNN, a geared rotation-identical CNNs (GRI-CNN) can be constructed by multiple TI-CNNs with the input vector rotated by a small step angle. Furthermore, a composed quantitatively identical CNN can also be constructed by combining multiple GRI-CNNs with various transformed input vectors.

1. "Transformationally Identical and Invariant Convolutional Neural Networks through Symmetric Element Operators” https://arxiv.org/abs/1806.03636 (June 2018)

2. “Transformationally Identical and Invariant Convolutional Neural Networks by Combining Symmetric Operations or Input Vectors” https://arxiv.org/abs/1807.11156 (July 2018)

3. "Geared Rotationally Identical and Invariant Convolutional Neural Network Systems" https://arxiv.org/abs/1808.01280 (August 2018)

Answer 4

I think max pooling can reserve translational and rotational invariances only for translations and rotations smaller than the stride size. If greater, no invariance