I am trying to revise the details of a Multi-layer Perceptron with a set of weight matrices $\mathcal W$ and a set of bias vectors $\mathbf b$. Here is the quadratic cost function I am using,

$$C(\mathcal W, \mathbf b) = \frac{1}{n} \sum_x(y-a^L)^2$$

I understand that in order to run (Stocastic) Gradient Descent we need the gradient of the cost function $\nabla C(\mathcal W, \mathbf b),$

$$\nabla C(\mathcal W, \mathbf b) = \Bigg[\frac{\partial C}{\partial w^1}, \frac{\partial C}{\partial b^1}, ... , \frac{\partial C}{\partial w^L}, \frac{\partial C}{\partial b^L}\Bigg]$$

Here is my question, for some layer $l$, is my derivation of $\frac{\partial C}{\partial w^l}$ correct (I am nervous about a negative sign)?

\begin{align*} \frac{\partial C}{\partial w^l} & = \frac{1}{n} \sum_x \Bigg( \frac{\partial (y-a^l)^2}{\partial (y-a^l)} \frac{\partial (y-a^l)}{\partial w^l} \Bigg) && \frac{\partial f(g(x))}{\partial (x)} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial x} \\ & = \frac{1}{n} \sum_x \Bigg( \frac{\partial (y-a^l)^2}{\partial (y-a^l)} \Big(\frac{\partial (y)}{\partial w^l} - \frac{\partial (a^l)}{\partial w^l} \Big)\Bigg) && \frac{\partial (f \pm g)}{\partial x} = \frac{\partial (f)}{\partial x} \pm \frac{\partial (g)}{\partial x} \\ & = \frac{1}{n} \sum_x \Bigg( \frac{\partial (y-a^l)^2}{\partial (y-a^l)} \Big(0 - \frac{\partial (a^l)}{\partial w^l} \Big)\Bigg) && \frac{\partial (y)}{\partial w^l} = 0 \\ & = \frac{1}{n} \sum_x \Bigg( \frac{\partial (y-a^l)^2}{\partial (y-a^l)} \Big(- \frac{\partial (a^l)}{\partial w^l} \Big)\Bigg) \\ & = \frac{1}{n} \sum_x \Bigg( - \frac{\partial (y-a^l)^2}{\partial (y-a^l)} \frac{\partial (a^l)}{\partial w^l} \Bigg) && \text{Brought the } (-) \text{ up front} \\ & = \frac{1}{n} \sum_x \Bigg( -\frac{\partial (y-a^l)^2}{\partial (y-a^l)} \Big(\frac{\partial (\sigma(z^l))}{\partial z^l} \frac{\partial z^l}{\partial w^l} \Big)\Bigg) && \text{Chain Rule} \\ & = \frac{1}{n} \sum_x \Bigg( -\frac{\partial (y-a^l)^2}{\partial (y-a^l)} \frac{\partial (\sigma(z^l))}{\partial z^l} \frac{\partial z^l}{\partial w^l} \Bigg) \end{align*}

Evaluating differentials,

\begin{align*} \frac{\partial (y-a^l)^2}{\partial (y-a^l)} & = 2(y-a^l) \\ \frac{\partial (\sigma(z^l))}{\partial z^l} & = \sigma'(z^l) && \sigma'(x) = \sigma(x)(1-\sigma(x)) \\ \frac{\partial z^l}{\partial w^l} & = \frac{\partial (w^l a^{l-1} + b^l)}{\partial w^l} = a^{l-1} \end{align*}

Therefore, $$ \frac{\partial C}{\partial w^l} = \frac{1}{n} \sum_x \Bigg( -\frac{\partial (y-a^l)^2}{\partial (y-a^l)} \frac{\partial (\sigma(z^l))}{\partial z^l} \frac{\partial z^l}{\partial w^l} \Bigg) = \frac{1}{n} \sum_x (-2(y-a^l)\sigma'(z^l)a^{l-1}) $$

I accumulate a negative sign in front which makes me nervous because I did not see it at other places (though I do understand the choice of a half in front of the quadratic cost function but I chose not to add it).

I did also find places which had a negative sign upfront like me: here and here.

So, is my derivation of $\frac{\partial C}{\partial w^l}$ correct?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.