Invariance in neural networks

Question

I have tried to read about tangent propagation in neural networks (although I guess it could be applicable to other methods) which is a procedure to create models that are invariant to certain transformation. So basically the argument goes like this:

Start with an input vector $x_{n}$. Then use a linear transformation (maybe a rotation or a translation) $s(x_{n},\xi)$ such that when $\xi=0$, it returns the same input: $s(x_{n},0)=x_{n}$. So when we apply a given transformation $s(x_{n},\xi)$ we move $x_{n}$ in some way (according to the value of $\xi$.) If we plot every new value for $x_{n}$ depending on our choice of $\xi$ then we can draw a curve $M$. Then we ask for the tangent vector of this curve in the original point $\xi = 0$, which is given by:

$$\tau_{n}=\displaystyle \frac{\partial s(x_{n},\xi)}{\partial \xi}\bigg|_{\xi=0}$$

At this point, I think I understand what is going on. However, I can't justify the following calculation:

Let's suppose I want to know the derivative of the output vector $y_{k}$ with respect to $\xi$ at the original point $\xi=0$. That's a fair question to ask since the output vector depends on the input vectors. However, Bishop's book (page 264) says that

$$\frac{\partial y_{k}}{\partial \xi}\bigg|_{\xi=0}=\sum_{i=1}^{D}\frac{\partial y_{k}}{\partial x_{i}}\frac{\partial x_{i}}{\partial \xi}\bigg|_{\xi=0}=\sum_{i=1}^{D}J_{ki}\tau_{i}$$

where $J_{ki}$ is the $k,i$ element of the Jacobian matrix.

As I understand it, we are replacing $x_{n}$ by its transformation $s(x_{n},\xi)$, so I think $y_{k}$ depends on $s(x_{n},\xi)$ and $s(x_{n},\xi)$ (as it indicates) depends on $x_{n}$ and $\xi$.

Actually, I don't understand how the expression above can be correct, since $\tau_{i}$ is defined in the first equation based on $s(x_{n},\xi)$, instead of $x_{i}$

What is the reason for this description of the change of $y_{k}$ when we change a bit $\xi$?

Note: I found out the original paper by Simard in which it's explained the general idea but unfortunately he doesn't carry out the calculation of $\frac{\partial y_{k}}{\partial \xi}\bigg|_{\xi=0}$, so no help from there.

Answer 1

The author is just abusing the notation a bit, but essentially he's applying the chain rule. $\mathbf{x} = (x_{1}, ..., x_{n})$ represents the input pattern to the network. Now, for each input pattern, new samples are generated by applying the transformation represented by $s(x_{n},\xi)$.

Let us denote these new patterns by $\mathbf{t} = s(\mathbf{x},\xi)$ (for transformed). Now, $\mathbf{y} = \mathbf{y}(\mathbf{t};\mathbf{w})$ is the mapping realized by the neural network. To see how the response of the network is affected by changes in the transformation parameter you apply the chain rule (I follow your component-wise decomposition),

$$\frac{\partial y_{k}}{\partial \xi} = \sum_{i=1}^{D} \frac{\partial y_{k}}{\partial t_{k}} \frac{\partial t_{k}}{\partial \xi} = \sum_{i=1}^{D} J_{ki} \tau_{i}$$

Here $\mathbf{x}$ is fixed, and you vary $\xi$. I guess he wanted to stress the fact that the components of the Jacobian are just the expressions obtained when talking about the backpropagation algorithm. It is just a concrete instance of the concept of directional derivative.