Activation at back propagation

Question

I need a short description about a part of a general backpropagation equation. Following this excellent article:

Output layer bp equation

$ \delta_{j}^L=\frac{\partial C}{\partial a_{j}^L}\sigma^{\prime}(z_{j}^L) $

Where:

$\frac{\partial C}{\partial a_{j}^L}$ is a partial derivative of a cost function. For example Quadratic Cost could be derived as $(a_{j}^L-y_{j})$ and applied to a vectors element-wise, that is clear.

Unclear part

But what is $\sigma^{\prime}(z_{j}^L)$ ?

• $\sigma$ is an activation function, for example SoftMax.
• $z_{j}^L=\sum_{\substack{k}}w_{jk}^La_{k}^{L-1}+b^L$ which is weighted activation input coming from a previous layer. That is clear too.

But this prime $\prime$ after activation function made me doubt that I fully understand this equation. I have several assumptions about a way to treat it in code:

• just use a weighted input for each neuron of a previous layer. But if it is true, why not just use $z_{j}^L$ ?
• use a partial derivative of activation function with respect to input value. But why not to use the same derivative notation which used at a declaration of the first part of the equation?

I hope I asked it clear, if not ask me to clarify anything in a comments. Thanks!

Answer 1

The prime $'$ is a common symbol for the derivative of a function. The equation you wrote is simply invokes the chain rule, i.e. $\frac{\partial f(g(x))}{\partial x} = \frac{\partial f(g(x))}{\partial g(x)} \frac{d g(x)} {d x}.$

Because $\delta_j^L = \frac{\partial C} {\partial z_j^L}$ and $a_j^L = \sigma(z_j^L)$, we have, $$ \delta_j^L = \frac{\partial C} {\partial a_j^L} \frac{d a_j^L}{d z_j^L} = \frac{\partial C} {\partial a_j^L} \sigma'(z_j^L), $$

where I simply invoked the prime convention to denote a derivative, i.e. $\sigma'(x) = \frac{d \sigma(x)} {d x}.$