Clarification needed on gradients in backpropagation

Question

I was going through this book "Practical Convolutional Neural Networks" and there under the backpropagation section, it demonstrates calculating the gradient for x and W for a single neuron with a sigmoid activation function.

z = 1/(1 + np.exp(-np.dot(W, x)))   # forward pass
dx = np.dot(W.T, z*(1-z))           # backward pass: local gradient for x
dW = np.outer(z*(1-z), x)           # backward pass: local gradient for W

I do understand why the gradient of x has np.dot in it. But I don't understand how the gradient for W has np.outer. A proper mathematical derivation corresponding to this would be really helpful. Thanks.

Answer 1

Assume there are multiple neurons, and each row of $W$ are weights of them, i.e. $W$ is of dimension $m\times n$, where we have $m$ neurons. Then, $z$ is of dimension $m\times 1$ and $x$ has dimension $n\times 1$. This is what's being assumed as far as I can see from the implementation.

If we wanted to find the derivative of loss with respect to $W_{ij}$, we'll have the following chain rule: $$\frac{\partial L}{\partial W_{ij}}=\frac{\partial L}{\partial z_i}\frac{\partial z_i}{\partial W_{ij}}$$

Here, $z_i$ is the output of the $i$-th neuron, and is $z_i=\sigma(h_i)$ where $h_i=\sum_{j=1}^n W_{ij}x_j$ So, $$\frac{\partial z_i}{\partial W_{ij}}=\frac{\partial z_i}{\partial h_i}\frac{\partial h_i}{\partial W_{ij}}=z_i(1-z_i)x_j$$

So, we have derivative matrix (call it $dW$) with $dW_{ij}=z_i(1-z_i)x_j$. This is exactly the outer product of vectors $z(1-z)$ (element-wise multiplication) and $x$.

This is not needed for a single neuron, because $z(1-z)$ will be just a scalar.