Given a neural network as following \begin{align*} &J = CE(y,\hat{y})=-\sum_i y_i log(\hat{y}_i)\\ &\hat{y} = softmax(z_2)\\ &z_2 = hW_2+b_2\\ &h = sigmoid(z_1)\\ &z_1 = xW_1+b_1 \end{align*} and $$\frac{\partial J}{\partial z_2}=\hat{y}-y=\delta_1$$ where $W_1 \in \mathbb{R}^{D_x \times H}$, $b_1 \in \mathbb{R}^H$, $W_2 \in \mathbb{R}^{H \times D_y}$ and $b_2 \in \mathbb{R}^{D_y}$. Derive the gradients of $J$ with respect to $h$, $W_2$.

Here's the correct solution: \begin{align*} &\frac{\partial J}{\partial h}=\delta_1 \frac{\partial z_2}{\partial h}=\delta_1 W_2^T\\ &\frac{\partial J}{\partial W_2}=\frac{\partial z_2}{\partial W_2} \frac{\partial J}{\partial z_2}=h^T \delta_1 \end{align*}

Q1: I have difficulties to understand the second dervitive with repsect to $W_2$, where $z_2 \in \mathbb{R}^{D_y}$ and $W_2 \in \mathbb{R}^{H \times D_y}$, the result $\frac{\partial z_2}{\partial W_2}$ to me should have a dimesion $(D_y,H \times D_y)$, $h^T$ doesn't satisfy this dimension.

Q2: why does the second derivative have an order of $h^T \delta_1$ not $\delta_1 h^T$ by following backpropagation such that $\frac{\partial J}{\partial W_2}=\frac{\partial J}{\partial z_2} \frac{\partial z_2}{\partial W_2}=\delta_1 h^T$?

Notes: $z_1$ and $z_2$ are calculated using array broadcasting in numpy, for instance the arithmetic is legit: array(4x3)+array(3)=array(4x3), The principle bechind the scence is that array(3) could be converted to array(4x3) via stacking. Details of broadcasting. [This is trivia]

  • $\begingroup$ Shouldnt $\partial z_2 / \partial W_2 $ have the dimension of $W_2$? $\endgroup$ – Nikolas Rieble Jul 11 '17 at 9:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.