2
$\begingroup$

In Glorot 2010 Section 4.2.1, the author states that in formula (6)

$$ Var\left[\dfrac{\partial Cost}{\partial s^i}\right] = Var\left[\dfrac{\partial Cost}{\partial s^d}\right] \prod_{i'=i}^{d}n_{i' +1}Var\left[W^{i'}\right] $$ where $s^i$ is the activation output from layer $i$ and $n_{i}$ is the number of neurons in layer $i$, where $s^i = z^i W^i + b^i$ are the linear transformation between layers.

Can anyone derive formula (6) from formula (2): $$ \dfrac{\partial Cost}{\partial s^i_k} = f'(s_k^i) W^{i+1}_{k,\dots} \dfrac{\partial Cost}{\partial s^{i+1}} $$

The problem here is that $W^{i+1}_{k,\dots}$ and $\dfrac{\partial Cost}{\partial s^{i+1}}$ are not independent to each other ($\dfrac{\partial Cost}{\partial s^{i+1}}$ is a function of the output layer.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.