Based on the (going deeper with convolutions) paper. The inception module consists of multiple branches, each branch consists of convolution layer and each layer has a different kernel size for example (1×1, 3×3, 5×5). The benefit of this model is to make your model choose the beast feature maps from these branches by concatenating them before next layer.

The problem with it the expensive computation because of the (3×3, 5×5) layers and might cover the optimal sparse structure issue. So they add before each layer the (1×1) conv layer to solve these two problems.

my question is:: How does the (1×1) conv layer reduce from the complexity and solve sparse structure issue???


Imagine you have an input tensor of size (32,32,512) where the dimensions correspond to (Height,Width,Channels) and imagine you have to apply a 128 different 5x5 convolutions to this tensor. This means you will need 128 filters of size (5,5,512) which is quite a lot of computation. Each filter needs to have a depth of 512 in order to account for the depth of the input tensor.

Now, instead of applying 5x5 convolutions directly, you apply 128 different 1x1 convolutions first. To do this you will need 128 filters of size (1,1,512) which is much less (about 25 times less) computation than applying 128 5x5 filters. This is because to do a 5x5 filter, you have to take 25*512 multiplications and add them together at each step, while to do a 1x1 filter you only do 512 multiplications at each step. After you passed the 1x1 convolutions you are left with a tensor of shape (32,32,128) which you can then pass onto the 5x5 convolutions. Having reduced the depth of the tensor from 512 to 128 your 5x5 convolutions will have much less work to do.

The net effect of the 1x1 convolutions is if you took each pixel in your input tensor and ran them through a (shared) 1 layer neural network to reduce their size. In the example above, regard each pixel of the input tensor as a 512 dimensional input to a fully connected neural network a 128 dimensional output. That's essentially what the 1x1 convolutions are doing. If your tensors have become so deep that they are unweildy (as happens in inception), then it might make sense to use 1x1 convlutions to reduce the depth of the tensors before applying larger convolutions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.