39
$\begingroup$

We define a bottleneck architecture as the type found in the ResNet paper where [two 3x3 conv layers] are replaced by [one 1x1 conv, one 3x3 conv, and another 1x1 conv layer]. enter image description here

I understand that the 1x1 conv layers are used as a form of dimension reduction (and restoration), which is explained in another post. However, I am unclear about why this structure as effective as the original layout.

Some good explanations might include: What stride length is used and at what layers? What are example input and output dimensions of each module? How are the 56x56 feature maps represented in the diagram above? Do the 64-d refer to the number of filters, why does this differ from the 256-d filters? How many weights or FLOPs are used at each layer?

Any discussion is greatly appreciated!

Improve this question
$\endgroup$
1
  • $\begingroup$ I think it may help with generalization and prevention of over fitting. But that is just a vague recollection. $\endgroup$
    – Chris
    Apr 4, 2016 at 4:18

3 Answers 3

Reset to default
11
$\begingroup$

The bottleneck architecture is used in very deep networks due to computational considerations.

To answer your questions:

  1. 56x56 feature maps are not represented in the above image. This block is taken from a ResNet with input size 224x224. 56x56 is the downsampled version of the input at some intermediate layer.

  2. 64-d refers to the number of feature maps(filters). The bottleneck architecture has 256-d, simply because it is meant for much deeper network, which possibly take higher resolution image as input and hence require more feature maps.

  3. Refer this figure for parameters of each bottleneck layer in ResNet 50.

Improve this answer
$\endgroup$
3
  • 4
    $\begingroup$ For future readers, I should mention that I think the 1x1 convs have stride=1 and pad=0, to preserve (WxH) of 56x56. Similarly, the 3x3 convs have stride=1 and pad=1 to preserve size as well. $\endgroup$
    – derekchen14
    May 12, 2017 at 20:59
  • 1
    $\begingroup$ Still I dont understand. It seems both of them have the similar amount of parameters, in that case I still dont understand the purpose of bootleneck layer. $\endgroup$
    – user570593
    Apr 2, 2018 at 3:24
  • 1
    $\begingroup$ I really think that the 2nd point in Newstein's answer is misleading. The 64-d or 256-d should refer to the number of channels of the input feature map — not the number of input feature maps. Consider the "bottleneck" block (the right of the figure) in the OP's question as an example: - 256-d means that we have a single input feature map with dimension n x n x 256. The 1x1, 64 in the figure means ` 64` filters, each is 1x1 and has 256 channels (1x1x256). - So here we can see that the convolution of a single filter (1x1x256) with an input feature map (n x n x 256) gi $\endgroup$
    – IgNite
    Oct 26, 2019 at 18:35
4
$\begingroup$

As far as I understand, illustration on the right shows that input to this block already has 256 features. So we are deep into some ResNet architecture and already created 256 features (we lost some w x h due to conv 3x3 before but gained features instead).

Still, calculating 256 channels (features) can take too much time, and authors proposed using 1x1 conv layer with stride 1 and padding 0 that keeps image's w x h as it was but in the meantime reduces depth, number of output channels to 64.

Thus, but using this bottleneck, at first layer you are shoving w x h x 256 element into conv 1x1 layer that will pass through just w x h x 64

Improve this answer
$\endgroup$
1
$\begingroup$

I just want to answer your questions:

What stride length is used and at what layers? Stride is not relevant in this discussion/figure. The "layers" are represented as rectangles and are 2d convolutional layers.

How are the 56x56 feature maps represented in the diagram above? Feature maps are the result of a conv. layer. So they should be positioned right after the rectangles (which are the convolutional layers). The dimension of the feature maps are not provided in the figure but can be calculated.

Do the 64-d refer to the number of filters, why does this differ from the 256-d filters? Yes, they indicate the number of filters/kernels used in this layer. A lower number means that fewer filters are used which means that the network has less capacity to represent/store information. By this mechanism, a bottleneck is obtained and it is assumed that the network learns to focus on the most relevant things and remove noise.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.