0
$\begingroup$

I need a short description about a part of a general backpropagation equation. Following this excellent article:

Output layer bp equation

$ \delta_{j}^L=\frac{\partial C}{\partial a_{j}^L}\sigma^{\prime}(z_{j}^L) $

Where:

$\frac{\partial C}{\partial a_{j}^L}$ is a partial derivative of a cost function. For example Quadratic Cost could be derived as $(a_{j}^L-y_{j})$ and applied to a vectors element-wise, that is clear.

Unclear part

But what is $\sigma^{\prime}(z_{j}^L)$ ?

  • $\sigma$ is an activation function, for example SoftMax.
  • $z_{j}^L=\sum_{\substack{k}}w_{jk}^La_{k}^{L-1}+b^L$ which is weighted activation input coming from a previous layer. That is clear too.

But this prime $\prime$ after activation function made me doubt that I fully understand this equation. I have several assumptions about a way to treat it in code:

  • just use a weighted input for each neuron of a previous layer. But if it is true, why not just use $z_{j}^L$ ?
  • use a partial derivative of activation function with respect to input value. But why not to use the same derivative notation which used at a declaration of the first part of the equation?

I hope I asked it clear, if not ask me to clarify anything in a comments. Thanks!

$\endgroup$
  • $\begingroup$ Your use of $\alpha$ is confusing. The article you linked defines $a$ as the activation of a node, i.e. $a_j^l = \sigma(z_j^l)$ which makes the equations make sense. But when people read your post, as it stands with the $\alpha$s, they will be confused until they actually go to your link and learn the article's naming conventions. $\endgroup$ – Bridgeburners Jun 5 '17 at 17:32
  • $\begingroup$ @Bridgeburners, I'm sorry I can't find what is confusing? I tried to use the same notation as the article. Do I have a typo? $\endgroup$ – I159 Jun 5 '17 at 17:38
  • $\begingroup$ You're using the alpha symbol ($\alpha$) and the $a$ symbol interchangeably. From what I can see, there is no $\alpha$ in the article you linked, only $a.$ It's confusing because, before going to the article, I thought you might have meant something else by $\alpha,$ but you simply meant $a.$ $\endgroup$ – Bridgeburners Jun 5 '17 at 17:44
1
$\begingroup$

The prime $'$ is a common symbol for the derivative of a function. The equation you wrote is simply invokes the chain rule, i.e. $\frac{\partial f(g(x))}{\partial x} = \frac{\partial f(g(x))}{\partial g(x)} \frac{d g(x)} {d x}.$

Because $\delta_j^L = \frac{\partial C} {\partial z_j^L}$ and $a_j^L = \sigma(z_j^L)$, we have, $$ \delta_j^L = \frac{\partial C} {\partial a_j^L} \frac{d a_j^L}{d z_j^L} = \frac{\partial C} {\partial a_j^L} \sigma'(z_j^L), $$

where I simply invoked the prime convention to denote a derivative, i.e. $\sigma'(x) = \frac{d \sigma(x)} {d x}.$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.