The following is a question regarding a residual block. Let $A^{[l-1]\{t\}}$ denote the activation of the $(l-1)^\mathrm{th}$ layer of a particular fully connected neural network, given the $t^\mathrm{th}$ mini batch of our training set. Suppose therefore that we have some activation $A^{[l-1]\{t\}}$ being propagated forward into an identity block contained by an $L$-layered fully connected neural network, where $L>l+1$. In this case, the next computation we would perform is $Z^{[l]\{t\}}=W^{[l]}A^{[l-1\{t\}}+B^{[l]}$, followed by $A^{[l]\{t\}}=g^{[l]}(Z^{[l]\{t\}})$ (where $g^{[l]}(z)$ denotes the non-linear activation function of the $l^\mathrm{th}$ layer of the network in question, applied element-wise to its input, $z$). This is in turn followed by $Z^{[l+1]\{t\}}=W^{[l+1]}A^{[l]\{t\}}+B^{[l+1]}$, and from here, if we suppose that $\tilde{Z}^{[l+1]\{t\}}=Z^{[l+1]\{t\}}+A^{[l-1]\{t\}}$, then $A^{[l+1]\{t\}}=g^{[l+1]}(\tilde{Z}^{[l+1]\{t\}})$. The rest of the forward pass continues from here. If we regard the act of a 'skip' in our network as the omission of a step we would execute, were the identity block not present, then, in our forward pass, we haven't performed a 'skip', per sé (although we have modified the computation of $A^{[l+1]\{t\}}$). However, when we perform our backward pass, once we've evaluated $\partial \tilde{Z}^{[l+1]\{t\}}$, we may pass $\partial \tilde{Z}^{[l+1]\{t\}}$ backwards into not only the adjacent layer (the $l^\mathrm{th}$ layer), but also the $(l-1)^\mathrm{th}$ layer (which is the layer preceding our identity block). We may then use $\partial \tilde{Z}^{[l+1]\{t\}}$ to train $W^{[l-1]}$ and $B^{[l-1]}$, instead of $\partial A^{[l-1]\{t\}}$ (which is what we'd usually use to train said parameters). And in this process, we've performed a 'skip', since we've omitted the use of $\partial A^{[l-1]\{t\}}$ to train $W^{[l-1]}$ and $B^{[l-1]}$ by instead using $\partial\tilde{Z}^{[l+1]\{t\}}$.

For some reason, the notion of a residual block didn't occur as naturally to me as it seems to have occurred to others, so I'm posting this question to confirm whether or not I'm understanding identity (and, more broadly, residual) blocks correctly. That being said, if anyone is able to confirm that what I've explained above, I'd be very thankful. If I need to add some clarity to certain portions of the question, please do ask. Thank you in advance.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.