Activation at back propagation

I need a short description about a part of a general backpropagation equation. Following this excellent article:

Output layer bp equation

$\delta_{j}^L=\frac{\partial C}{\partial a_{j}^L}\sigma^{\prime}(z_{j}^L)$

Where:

$\frac{\partial C}{\partial a_{j}^L}$ is a partial derivative of a cost function. For example Quadratic Cost could be derived as $(a_{j}^L-y_{j})$ and applied to a vectors element-wise, that is clear.

Unclear part

But what is $\sigma^{\prime}(z_{j}^L)$ ?

• $\sigma$ is an activation function, for example SoftMax.
• $z_{j}^L=\sum_{\substack{k}}w_{jk}^La_{k}^{L-1}+b^L$ which is weighted activation input coming from a previous layer. That is clear too.

But this prime $\prime$ after activation function made me doubt that I fully understand this equation. I have several assumptions about a way to treat it in code:

• just use a weighted input for each neuron of a previous layer. But if it is true, why not just use $z_{j}^L$ ?
• use a partial derivative of activation function with respect to input value. But why not to use the same derivative notation which used at a declaration of the first part of the equation?

I hope I asked it clear, if not ask me to clarify anything in a comments. Thanks!

• Your use of $\alpha$ is confusing. The article you linked defines $a$ as the activation of a node, i.e. $a_j^l = \sigma(z_j^l)$ which makes the equations make sense. But when people read your post, as it stands with the $\alpha$s, they will be confused until they actually go to your link and learn the article's naming conventions. – Bridgeburners Jun 5 '17 at 17:32
• @Bridgeburners, I'm sorry I can't find what is confusing? I tried to use the same notation as the article. Do I have a typo? – I159 Jun 5 '17 at 17:38
• You're using the alpha symbol ($\alpha$) and the $a$ symbol interchangeably. From what I can see, there is no $\alpha$ in the article you linked, only $a.$ It's confusing because, before going to the article, I thought you might have meant something else by $\alpha,$ but you simply meant $a.$ – Bridgeburners Jun 5 '17 at 17:44

The prime $'$ is a common symbol for the derivative of a function. The equation you wrote is simply invokes the chain rule, i.e. $\frac{\partial f(g(x))}{\partial x} = \frac{\partial f(g(x))}{\partial g(x)} \frac{d g(x)} {d x}.$
Because $\delta_j^L = \frac{\partial C} {\partial z_j^L}$ and $a_j^L = \sigma(z_j^L)$, we have, $$\delta_j^L = \frac{\partial C} {\partial a_j^L} \frac{d a_j^L}{d z_j^L} = \frac{\partial C} {\partial a_j^L} \sigma'(z_j^L),$$
where I simply invoked the prime convention to denote a derivative, i.e. $\sigma'(x) = \frac{d \sigma(x)} {d x}.$