I was reading the paper Deep Residual Learning for Image Recognition and I had difficulties understanding with 100% certainty what a residual block entails computationally. Reading their paper they have figure 2:

enter image description here

which illustrates what a Residual Block is suppose to be. Is the computation of a residual block simply the same as:

$$ \mathbf{y} = \sigma( W_2 \sigma( W_1 \mathbf{x} + b_1 ) + b_2 + \mathbf{x} )$$

Or is it something else?

In other words maybe to try to match the paper's notation, is:

$$ \mathcal F(x) + x = \left[ W_2 \sigma( W_1 \mathbf{x} + b_1 ) + b_2 \right] + \mathbf{x}$$

is that true?

Notice that after the circle summation, the word ReLU appears on the paper, so the output of a Residual Block (which I denoted by $\mathbf{y}$) should be:

$$ \sigma( \mathcal F(x) + x ) = \sigma( \left[ W_2 \sigma( W_1 \mathbf{x} + b_1 ) + b_2 \right] + \mathbf{x} )$$

with one additional ReLU non-linearity $\sigma$.

  • $\begingroup$ is x is positive relu(x)=x $\endgroup$
    – Ray Tayek
    Nov 11 '17 at 11:56

Yes that's true, you can take a look at their caffe model to see how it is implemented.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.