# Is my step by step derivation of quadratic cost function's (Neural Networks) partial derivative with respect to some weights matrix correct?

I am trying to revise the details of a Multi-layer Perceptron with a set of weight matrices $$\mathcal W$$ and a set of bias vectors $$\mathbf b$$. Here is the quadratic cost function I am using,

$$C(\mathcal W, \mathbf b) = \frac{1}{n} \sum_x(y-a^L)^2$$

I understand that in order to run (Stocastic) Gradient Descent we need the gradient of the cost function $$\nabla C(\mathcal W, \mathbf b),$$

$$\nabla C(\mathcal W, \mathbf b) = \Bigg[\frac{\partial C}{\partial w^2}, \frac{\partial C}{\partial b^2}, ... , \frac{\partial C}{\partial w^L}, \frac{\partial C}{\partial b^L}\Bigg]$$

Here is my question, for some layer $$l$$, is my derivation of $$\frac{\partial C}{\partial w^l}$$ correct (I am nervous about a negative sign)?

\begin{align*} \frac{\partial C}{\partial w^l} & = \frac{1}{n} \sum_x \Bigg( \frac{\partial (y-a^l)^2}{\partial (y-a^l)} \frac{\partial (y-a^l)}{\partial w^l} \Bigg) && \frac{\partial f(g(x))}{\partial (x)} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial x} \\ & = \frac{1}{n} \sum_x \Bigg( \frac{\partial (y-a^l)^2}{\partial (y-a^l)} \Big(\frac{\partial (y)}{\partial w^l} - \frac{\partial (a^l)}{\partial w^l} \Big)\Bigg) && \frac{\partial (f \pm g)}{\partial x} = \frac{\partial (f)}{\partial x} \pm \frac{\partial (g)}{\partial x} \\ & = \frac{1}{n} \sum_x \Bigg( \frac{\partial (y-a^l)^2}{\partial (y-a^l)} \Big(0 - \frac{\partial (a^l)}{\partial w^l} \Big)\Bigg) && \frac{\partial (y)}{\partial w^l} = 0 \\ & = \frac{1}{n} \sum_x \Bigg( \frac{\partial (y-a^l)^2}{\partial (y-a^l)} \Big(- \frac{\partial (a^l)}{\partial w^l} \Big)\Bigg) \\ & = \frac{1}{n} \sum_x \Bigg( - \frac{\partial (y-a^l)^2}{\partial (y-a^l)} \frac{\partial (a^l)}{\partial w^l} \Bigg) && \text{Brought the } (-) \text{ up front} \\ & = \frac{1}{n} \sum_x \Bigg( -\frac{\partial (y-a^l)^2}{\partial (y-a^l)} \Big(\frac{\partial (\sigma(z^l))}{\partial z^l} \frac{\partial z^l}{\partial w^l} \Big)\Bigg) && \text{Chain Rule} \\ & = \frac{1}{n} \sum_x \Bigg( -\frac{\partial (y-a^l)^2}{\partial (y-a^l)} \frac{\partial (\sigma(z^l))}{\partial z^l} \frac{\partial z^l}{\partial w^l} \Bigg) \end{align*}

Evaluating differentials,

\begin{align*} \frac{\partial (y-a^l)^2}{\partial (y-a^l)} & = 2(y-a^l) \\ \frac{\partial (\sigma(z^l))}{\partial z^l} & = \sigma'(z^l) && \sigma'(x) = \sigma(x)(1-\sigma(x)) \\ \frac{\partial z^l}{\partial w^l} & = \frac{\partial (w^l a^{l-1} + b^l)}{\partial w^l} = a^{l-1} \end{align*}

Therefore, $$\frac{\partial C}{\partial w^l} = \frac{1}{n} \sum_x \Bigg( -\frac{\partial (y-a^l)^2}{\partial (y-a^l)} \frac{\partial (\sigma(z^l))}{\partial z^l} \frac{\partial z^l}{\partial w^l} \Bigg) = \frac{1}{n} \sum_x (-2(y-a^l)\sigma'(z^l)a^{l-1})$$

I accumulate a negative sign in front which makes me nervous because I did not see it at other places (though I do understand the choice of a half in front of the quadratic cost function but I chose not to add it).

I did also find places which had a negative sign upfront like me: here and here.

So, is my derivation of $$\frac{\partial C}{\partial w^l}$$ correct?

• Can you link to a source that shows how to get a derivative of an equation with a summation sign? Jan 17, 2021 at 2:58
• It is not any different than normal polynomial derivatives. e.g., let $f(x) = \sum_{i=0}^{2} x^i = x^2 + x^1 + x^0 = x^2 + x + 1$. In this case, $\frac{df}{dx} = 2x + 1$. Jan 17, 2021 at 5:23
• You maybe interested in Neural Networks: The Backward Pass. Jan 17, 2021 at 5:25
• So I basically have to write out the summation as the "long-form" equation? Jan 17, 2021 at 9:10
• I am not sure about all things that can be written as a summation but for polynomials you don't have to write out the long form, you just have to imagine it. Though, you can write it out if it helps you in certain cases but it may not be always feasible. Consider: $f(x) = \sum_{i=1}^{2^{100}} x^i$. Here we know that $\frac{df}{dx} = \sum_{i=1}^{2^{100}} ix^{i-1} = \left(\sum_{i=2}^{2^{100}} ix^{i-1} \right) + 1$. Jan 17, 2021 at 22:43