1
$\begingroup$

I am trying to revise the details of a Multi-layer Perceptron with a set of weight matrices $\mathcal W$ and a set of bias vectors $\mathbf b$. Here is the quadratic cost function I am using,

$$C(\mathcal W, \mathbf b) = \frac{1}{n} \sum_x(y-a^L)^2$$

I understand that in order to run (Stocastic) Gradient Descent we need the gradient of the cost function $\nabla C(\mathcal W, \mathbf b),$

$$\nabla C(\mathcal W, \mathbf b) = \Bigg[\frac{\partial C}{\partial w^2}, \frac{\partial C}{\partial b^2}, ... , \frac{\partial C}{\partial w^L}, \frac{\partial C}{\partial b^L}\Bigg]$$

Here is my question, for some layer $l$, is my derivation of $\frac{\partial C}{\partial w^l}$ correct (I am nervous about a negative sign)?

\begin{align*} \frac{\partial C}{\partial w^l} & = \frac{1}{n} \sum_x \Bigg( \frac{\partial (y-a^l)^2}{\partial (y-a^l)} \frac{\partial (y-a^l)}{\partial w^l} \Bigg) && \frac{\partial f(g(x))}{\partial (x)} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial x} \\ & = \frac{1}{n} \sum_x \Bigg( \frac{\partial (y-a^l)^2}{\partial (y-a^l)} \Big(\frac{\partial (y)}{\partial w^l} - \frac{\partial (a^l)}{\partial w^l} \Big)\Bigg) && \frac{\partial (f \pm g)}{\partial x} = \frac{\partial (f)}{\partial x} \pm \frac{\partial (g)}{\partial x} \\ & = \frac{1}{n} \sum_x \Bigg( \frac{\partial (y-a^l)^2}{\partial (y-a^l)} \Big(0 - \frac{\partial (a^l)}{\partial w^l} \Big)\Bigg) && \frac{\partial (y)}{\partial w^l} = 0 \\ & = \frac{1}{n} \sum_x \Bigg( \frac{\partial (y-a^l)^2}{\partial (y-a^l)} \Big(- \frac{\partial (a^l)}{\partial w^l} \Big)\Bigg) \\ & = \frac{1}{n} \sum_x \Bigg( - \frac{\partial (y-a^l)^2}{\partial (y-a^l)} \frac{\partial (a^l)}{\partial w^l} \Bigg) && \text{Brought the } (-) \text{ up front} \\ & = \frac{1}{n} \sum_x \Bigg( -\frac{\partial (y-a^l)^2}{\partial (y-a^l)} \Big(\frac{\partial (\sigma(z^l))}{\partial z^l} \frac{\partial z^l}{\partial w^l} \Big)\Bigg) && \text{Chain Rule} \\ & = \frac{1}{n} \sum_x \Bigg( -\frac{\partial (y-a^l)^2}{\partial (y-a^l)} \frac{\partial (\sigma(z^l))}{\partial z^l} \frac{\partial z^l}{\partial w^l} \Bigg) \end{align*}

Evaluating differentials,

\begin{align*} \frac{\partial (y-a^l)^2}{\partial (y-a^l)} & = 2(y-a^l) \\ \frac{\partial (\sigma(z^l))}{\partial z^l} & = \sigma'(z^l) && \sigma'(x) = \sigma(x)(1-\sigma(x)) \\ \frac{\partial z^l}{\partial w^l} & = \frac{\partial (w^l a^{l-1} + b^l)}{\partial w^l} = a^{l-1} \end{align*}

Therefore, $$ \frac{\partial C}{\partial w^l} = \frac{1}{n} \sum_x \Bigg( -\frac{\partial (y-a^l)^2}{\partial (y-a^l)} \frac{\partial (\sigma(z^l))}{\partial z^l} \frac{\partial z^l}{\partial w^l} \Bigg) = \frac{1}{n} \sum_x (-2(y-a^l)\sigma'(z^l)a^{l-1}) $$

I accumulate a negative sign in front which makes me nervous because I did not see it at other places (though I do understand the choice of a half in front of the quadratic cost function but I chose not to add it).

I did also find places which had a negative sign upfront like me: here and here.

So, is my derivation of $\frac{\partial C}{\partial w^l}$ correct?

$\endgroup$
6
  • $\begingroup$ Can you link to a source that shows how to get a derivative of an equation with a summation sign? $\endgroup$ Jan 17 at 2:58
  • $\begingroup$ It is not any different than normal polynomial derivatives. e.g., let $f(x) = \sum_{i=0}^{2} x^i = x^2 + x^1 + x^0 = x^2 + x + 1$. In this case, $\frac{df}{dx} = 2x + 1$. $\endgroup$
    – scribe
    Jan 17 at 5:23
  • 1
    $\begingroup$ You maybe interested in Neural Networks: The Backward Pass. $\endgroup$
    – scribe
    Jan 17 at 5:25
  • $\begingroup$ So I basically have to write out the summation as the "long-form" equation? $\endgroup$ Jan 17 at 9:10
  • $\begingroup$ I am not sure about all things that can be written as a summation but for polynomials you don't have to write out the long form, you just have to imagine it. Though, you can write it out if it helps you in certain cases but it may not be always feasible. Consider: $f(x) = \sum_{i=1}^{2^{100}} x^i$. Here we know that $\frac{df}{dx} = \sum_{i=1}^{2^{100}} ix^{i-1} = \left(\sum_{i=2}^{2^{100}} ix^{i-1} \right) + 1$. $\endgroup$
    – scribe
    Jan 17 at 22:43
1
$\begingroup$

Alright, I will answer my own question since I feel more confident after doing some more research.

Is my step by step derivation of quadratic cost function's (Neural Networks) partial derivative with respect to some weights matrix correct?

Yes, It is. Though, the notation maybe sloppy.

I accumulate a negative sign in front which makes me nervous because I did not see it at other places

Sometimes, other derivations online distribute the negative sign and rewrite the final result in such a way that does not start with a negative sign at front. I missed the distribution and that made me feel nervous about me having a negative sign at front.


Finally, Neural Networks: The Backward Pass has the same derivation with a little less sloppy notation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.