Neural network with a single hidden layer of logistic units being used for a multi–class classification problem:

\begin{align} h &= \sigma (W^{(1)} x+b^{(1)}) \\[5pt] \hat y &= {\rm softmax}(W^{(2)}h + b^{(2)}) \end{align}

and trained using the cross–entropy error:

$$ C(y,\hat y) = -\sum_i y_i \log \hat y_i $$

I need to find the gradients of the error with respect to the parameters in the first layer, i.e., the layer closest to the input. The output target $y$ is a one-hot representation.

Was given this additional info: $$ \frac{\partial C}{\partial z} = y - \hat y $$ where $$ z = W^{(2)}h + b^{(2)} $$


1 Answer 1


Use the chain rule,

$$ \frac{\partial C}{\partial W_1} = \frac{\partial C}{\partial z} \frac{\partial z}{\partial h} \frac{\partial h}{\partial a} \frac{\partial a}{\partial W_1} , $$ where
$$ a = W_1x+b_1 . $$

$$ \frac{\partial C}{\partial z} = y-\hat{y}, $$

$$ \frac{\partial z}{\partial h} = W_2, $$

$$ \frac{\partial h}{\partial a} = \sigma(a)(1 - \sigma(a)), $$

$$ \frac{\partial a}{\partial W_1} = x, $$ so $$ \frac{\partial C}{\partial W_1} = (y-\hat{y})W_2\sigma(a)(1 - \sigma(a))x = (y-\hat{y})W_2h(1 - h)x. $$

You could take a look at Pattern Recognition and Machine Learning Section 5.3 for details.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.