# Invariance in neural networks

I have tried to read about tangent propagation in neural networks (although I guess it could be applicable to other methods) which is a procedure to create models that are invariant to certain transformation. So basically the argument goes like this:

Start with an input vector $x_{n}$. Then use a linear transformation (maybe a rotation or a translation) $s(x_{n},\xi)$ such that when $\xi=0$, it returns the same input: $s(x_{n},0)=x_{n}$. So when we apply a given transformation $s(x_{n},\xi)$ we move $x_{n}$ in some way (according to the value of $\xi$.) If we plot every new value for $x_{n}$ depending on our choice of $\xi$ then we can draw a curve $M$. Then we ask for the tangent vector of this curve in the original point $\xi = 0$, which is given by:

$$\tau_{n}=\displaystyle \frac{\partial s(x_{n},\xi)}{\partial \xi}\bigg|_{\xi=0}$$

At this point, I think I understand what is going on. However, I can't justify the following calculation:

Let's suppose I want to know the derivative of the output vector $y_{k}$ with respect to $\xi$ at the original point $\xi=0$. That's a fair question to ask since the output vector depends on the input vectors. However, Bishop's book (page 264) says that

$$\frac{\partial y_{k}}{\partial \xi}\bigg|_{\xi=0}=\sum_{i=1}^{D}\frac{\partial y_{k}}{\partial x_{i}}\frac{\partial x_{i}}{\partial \xi}\bigg|_{\xi=0}=\sum_{i=1}^{D}J_{ki}\tau_{i}$$

where $J_{ki}$ is the $k,i$ element of the Jacobian matrix.

As I understand it, we are replacing $x_{n}$ by its transformation $s(x_{n},\xi)$, so I think $y_{k}$ depends on $s(x_{n},\xi)$ and $s(x_{n},\xi)$ (as it indicates) depends on $x_{n}$ and $\xi$.

Actually, I don't understand how the expression above can be correct, since $\tau_{i}$ is defined in the first equation based on $s(x_{n},\xi)$, instead of $x_{i}$

What is the reason for this description of the change of $y_{k}$ when we change a bit $\xi$?

Note: I found out the original paper by Simard in which it's explained the general idea but unfortunately he doesn't carry out the calculation of $\frac{\partial y_{k}}{\partial \xi}\bigg|_{\xi=0}$, so no help from there.

• I think you are right, that does not make sense. What is the original text you are using? Commented Apr 25, 2013 at 6:50
• Sorry. I forgot to add that detail. Bishop's "Pattern classification and machine learning" I also ran into a document that described an application using tangent propagation in which the chain rule is used with $s$ instead of $x_{i}$ but the goal was a bit different, so I can't be sure about the correctness of that document either. Commented Apr 25, 2013 at 7:02

The author is just abusing the notation a bit, but essentially he's applying the chain rule. $\mathbf{x} = (x_{1}, ..., x_{n})$ represents the input pattern to the network. Now, for each input pattern, new samples are generated by applying the transformation represented by $s(x_{n},\xi)$.
Let us denote these new patterns by $\mathbf{t} = s(\mathbf{x},\xi)$ (for transformed). Now, $\mathbf{y} = \mathbf{y}(\mathbf{t};\mathbf{w})$ is the mapping realized by the neural network. To see how the response of the network is affected by changes in the transformation parameter you apply the chain rule (I follow your component-wise decomposition),
$$\frac{\partial y_{k}}{\partial \xi} = \sum_{i=1}^{D} \frac{\partial y_{k}}{\partial t_{k}} \frac{\partial t_{k}}{\partial \xi} = \sum_{i=1}^{D} J_{ki} \tau_{i}$$
Here $\mathbf{x}$ is fixed, and you vary $\xi$. I guess he wanted to stress the fact that the components of the Jacobian are just the expressions obtained when talking about the backpropagation algorithm. It is just a concrete instance of the concept of directional derivative.
• That makes so much sense, although can we still consider the elements $\frac{\partial y_{k}}{\partial t_{k}}$ as an element of the Jacobian? Commented Apr 30, 2013 at 3:24
• yes, they are the elements of the jacobian, because in the limit of $\xi = 0$ the $t = x$. That would be a good motivation to "abuse" the notation. Commented Apr 30, 2013 at 6:43