What is the difference between invariance to translation, covariance to translation and equivariance to translation?

Question

I get stuck at understanding the difference between invariance to translation, covariance to translation and equivariance to translation in the context of of convolutional neural network.

What does it mean :

1. Convolution is equivariant to translation ?
2. Convolution is convariant to translation ?

What is the difference between covariance and equivariance ?

3. Are convolutional layers equivariant to translation ?

Answer 1

There are two schools of thought when it comes to definitin of equivariance, covariance, invariance, and same-equivariance.

• Covariance is a concept often used in physics and is the same term as equivariance. Both are used when applying the transformation $\pi$ on the input of the function $f$ can be achieved by appying another transformation $\psi$ on the output of the function:
  $f(\pi(x))=\psi(f(x))$

• Same-equivariance is an especial case of equivariance when $\psi=\pi$ (in some literature, same-equivariance is termed as equivariance, and instead, equivariance is termed covariance): $f(\pi(x))=\pi(f(x))$

• Invariance is another especial case when the transformation $\psi$ is the identity function ($\psi=\mathbb{1}$)
  $f(\pi(x))=f(x)$

Based on the above definitions:

• convolution is "equivariant" to translation
• convolution is also "same-equivariant" to translation,
• and since covariance is just another term for the same concept, convolution is "covariant" to translation.

Same is true for convolutional layers.