1
$\begingroup$

I was going through this book "Practical Convolutional Neural Networks" and there under the backpropagation section, it demonstrates calculating the gradient for x and W for a single neuron with a sigmoid activation function.

z = 1/(1 + np.exp(-np.dot(W, x)))   # forward pass
dx = np.dot(W.T, z*(1-z))           # backward pass: local gradient for x
dW = np.outer(z*(1-z), x)           # backward pass: local gradient for W

I do understand why the gradient of x has np.dot in it. But I don't understand how the gradient for W has np.outer. A proper mathematical derivation corresponding to this would be really helpful. Thanks.

$\endgroup$
1
$\begingroup$

Assume there are multiple neurons, and each row of $W$ are weights of them, i.e. $W$ is of dimension $m\times n$, where we have $m$ neurons. Then, $z$ is of dimension $m\times 1$ and $x$ has dimension $n\times 1$. This is what's being assumed as far as I can see from the implementation.

If we wanted to find the derivative of loss with respect to $W_{ij}$, we'll have the following chain rule: $$\frac{\partial L}{\partial W_{ij}}=\frac{\partial L}{\partial z_i}\frac{\partial z_i}{\partial W_{ij}}$$

Here, $z_i$ is the output of the $i$-th neuron, and is $z_i=\sigma(h_i)$ where $h_i=\sum_{j=1}^n W_{ij}x_j$ So, $$\frac{\partial z_i}{\partial W_{ij}}=\frac{\partial z_i}{\partial h_i}\frac{\partial h_i}{\partial W_{ij}}=z_i(1-z_i)x_j$$

So, we have derivative matrix (call it $dW$) with $dW_{ij}=z_i(1-z_i)x_j$. This is exactly the outer product of vectors $z(1-z)$ (element-wise multiplication) and $x$.

This is not needed for a single neuron, because $z(1-z)$ will be just a scalar.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.