3
$\begingroup$

I was reading about resnet at this link. This link and others say that residual block skips 1-layer, but then all of them show a diagram where there is an additional weight layer (i think it can be also called conv layer) that seems to be skipped beyond just weight+relu of the skipped layer. In this diagram taken from above link, you can see the input x to the block is fed after the 2nd weight layer not before that. enter image description here

  • Why is that when the diagram is talking about only skipping 1-layer, it's showing skip-connection after relu? Isn't that part of the second conv+relu layer?

  • I've seen the input/output feature map used. Is the input feature map same as what is shown by 'x'?

  • isn't weight layer mean the same thing as performing conv using a filter?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

The original paper is quite readable and should answer most of your question. There are two reasons:

1) In section 3.1, the paper claims:

The form of the residual function \mathcal{F} is flexible. Experiments in this paper involve a function F that has two or three layers (Fig. 5), while more layers are possible. But if F has only a single layer, Eqn.(1) is similar to a linear layer: y = W1 x + x, for which we have not observed advantages.

2) For blocks, you need to ensure input/output shapes are the same: enter image description here

Notice that the input is 256-d, then there is a downsampling 1x1 convolution and then an upsampling back to 256-d. When you add the input $x$ to the output $F(x,W_i)$ of a given layer $i$, the result looks like:

$y = F(x,W_i)+x,$

which only makes sense if the dimensions of $F(x,W_i)$ and $x$ are the same. In the above example, you could add residual connections between each pair of successive layers, but then you'd have to adjust the dimensions, for example by using another matrix $W'_i$:

$y = F(x,W_i)+W'_ix.$

This has a disadvantage of massively increasing the total number of learned weights, and also diverts from the original intention of allowing an identity transformation, so that the network can "do nearly nothing" at any given block.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ this actually doesn't answer as it assumes that you're operating on the 'bottleneck block' (see fig 5 right-side in the linked paper) not the original residual block. for bottleneck block you need 1x1 layer around 3x3 layer to reduce/restore channels. but for the original residual block (my diagram in the OP), the channels are same even after the first 3x3 conv, but resnet still skips around 1.5 layers (i say 1.5 becuase it's more it's 2 layers of conv but 1 layer of relu).....(Contd..) $\endgroup$
    – Joe Black
    Apr 13, 2020 at 21:05
  • $\begingroup$ my question was "Why is that when the diagram is talking about only skipping 1-layer, it's showing skip-connection after relu?" so why doesn't it skip only 1 layer or 2layers and explicitly state that? $\endgroup$
    – Joe Black
    Apr 13, 2020 at 21:05
  • $\begingroup$ also, why doesn't it show BatchNorm layer after each Conv? $\endgroup$
    – Joe Black
    Apr 13, 2020 at 21:06
  • $\begingroup$ @JoeBlack: Ah. In that case, this remark from the paper (section 3.1) might shed some light (see edit above). The reasoning is a bit sketchy still, as the comparison to a linear layer seems a bit weird. $\endgroup$
    – Alex R.
    Apr 13, 2020 at 23:59
  • $\begingroup$ Batch norm occurs after the convolution, but before the activation. They mention this in the paper and they seem to abstract that away in the layer diagrams, specifying only the activations. $\endgroup$
    – Alex R.
    Apr 14, 2020 at 0:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.