In Glorot 2010 Section 4.2.1, the author states that in formula (6)

$$ Var\left[\dfrac{\partial Cost}{\partial s^i}\right] = Var\left[\dfrac{\partial Cost}{\partial s^d}\right] \prod_{i'=i}^{d}n_{i' +1}Var\left[W^{i'}\right] $$ where $s^i$ is the activation output from layer $i$ and $n_{i}$ is the number of neurons in layer $i$, where $s^i = z^i W^i + b^i$ are the linear transformation between layers.

Can anyone derive formula (6) from formula (2): $$ \dfrac{\partial Cost}{\partial s^i_k} = f'(s_k^i) W^{i+1}_{k,\dots} \dfrac{\partial Cost}{\partial s^{i+1}} $$

The problem here is that $W^{i+1}_{k,\dots}$ and $\dfrac{\partial Cost}{\partial s^{i+1}}$ are not independent to each other ($\dfrac{\partial Cost}{\partial s^{i+1}}$ is a function of the output layer.)



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.