I am new to neural networks. I am studying back propagation and saw different references. for a layer $k$, some references state that the error $\delta_j^k$ for neuron $j$ at $k$th layer is

$$ \delta_j^k = \dfrac{\partial E}{\partial a_j^k} $$

while some other references state

$$ \delta_j^k = \dfrac{\partial E}{\partial z_j^k} $$ where $z^k = w^l a^{(l-1)} + b^k$. Andrew Ng in his courses introduced this as $$ \delta^k = (W^{(k+1)})^T \delta^{(k+1)} .* \sigma^{'}(z^{(k)}) $$ that made me confused. Which one is true?


1 Answer 1


The answer to your question very much depends on the resource that you are using (i.e. there is no real right or wrong).

When using the notation $\boldsymbol{z}^{(l)} = \boldsymbol{W}^{(l)} \boldsymbol{a}^{(l-1)} + \boldsymbol{b}^{(l)}$ and $\boldsymbol{a}^{(l)} = \phi\bigl(\boldsymbol{z}^{(l)}\bigr)$, the error given by $$\boldsymbol{\delta}^{(l)} = {\boldsymbol{W}^{(l+1)}}^\mathsf{T} \boldsymbol{\delta}^{(l+1)} \odot \phi'\bigl(\boldsymbol{z}^{(l)}\bigr)$$ is the derivative w.r.t. the pre-activations, i.e. $\frac{\partial E}{\partial \boldsymbol{z}^{(l)}}$. (Note: I took the liberty to use $\phi$ for the activation function and $\odot$ to denote the Hadamard (i.e. element-wise) product.)

When considering the gradient w.r.t. the activations, i.e. $\frac{\partial E}{\partial \boldsymbol{a}^{(l)}}$, the error would be $$\boldsymbol{d}^{(l)} = {\boldsymbol{W}^{(l)}}^\mathsf{T} \boldsymbol{d}^{(l+1)} \odot \phi'\bigl(\boldsymbol{z}^{(l)}\bigr).$$ The difference is subtle, but the recursion aligns the weights differently.

However, you will rarely find the latter expression, because (almost) everyone uses the gradients w.r.t. the pre-activations. For some intuition as to why that is, I refer to this answer I gave to another question.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.