# What does 1x1 convolution mean in a neural network?

I am currently doing the Udacity Deep Learning Tutorial. In Lesson 3, they talk about a 1x1 convolution. This 1x1 convolution is used in Google Inception Module. I'm having trouble understanding what is a 1x1 convolution.

I have also seen this post by Yann Lecun.

Could someone kindly explain this to me?

• Also see a related question
– gkcn
Nov 1, 2016 at 10:57
• Here's a blog post on these modules that went into detail on the 1x1 convolutions: hackathonprojects.wordpress.com/2016/09/25/… Nov 15, 2016 at 6:46
• To put it simply, the whole idea is to reduce the number of channels in 3D volume, thus speeding up the computation. Mar 7, 2018 at 2:15
• 1x1 conv creates channel-wise dependencies with a negligible cost. This is especially exploited in depthwise-separable convolutions. Nobody said anything about this but I'm writing this as a comment since I don't have enough reputation here. Oct 29, 2021 at 13:12
• It is basically to average (or reduce) the input data (say $C*H*W$) across its channels (i.e., $C$). Convolution with one $1x1$ filter generates one average result in shape $H*W$. The $1x1$ filter is actually a vector of length $C$. When you have $F$ $1x1$ filters, you get $F$ averages. That means, your output data shape is $F*H*W$. Now your input $C*H*W$ becomes $F*H*W$ output. Jan 10 at 1:03

Suppose that I have a conv layer which outputs an $$(N, F, H, W)$$ shaped tensor where:

• $$N$$ is the batch size
• $$F$$ is the number of convolutional filters
• $$H, W$$ are the spatial dimensions

Suppose the input is fed into a conv layer with $$F_1$$ 1x1 filters, zero padding and stride 1. Then the output of this 1x1 conv layer will have shape $$(N, F_1, H , W)$$.

So 1x1 conv filters can be used to change the dimensionality in the filter space. If $$F_1 > F$$ then we are increasing dimensionality, if $$F_1 < F$$ we are decreasing dimensionality, in the filter dimension.

Indeed, in the Google Inception article Going Deeper with Convolutions, they state (bold is mine, not by original authors):

One big problem with the above modules, at least in this naive form, is that even a modest number of 5x5 convolutions can be prohibitively expensive on top of a convolutional layer with a large number of filters.

This leads to the second idea of the proposed architecture: judiciously applying dimension reductions and projections wherever the computational requirements would increase too much otherwise. This is based on the success of embeddings: even low dimensional embeddings might contain a lot of information about a relatively large image patch...1x1 convolutions are used to compute reductions before the expensive 3x3 and 5x5 convolutions. Besides being used as reductions, they also include the use of rectified linear activation which makes them dual-purpose.

So in the Inception architecture, we use the 1x1 convolutional filters to reduce dimensionality in the filter dimension. As I explained above, these 1x1 conv layers can be used in general to change the filter space dimensionality (either increase or decrease) and in the Inception architecture we see how effective these 1x1 filters can be for dimensionality reduction, explicitly in the filter dimension space, not the spatial dimension space.

Perhaps there are other interpretations of 1x1 conv filters, but I prefer this explanation, especially in the context of the Google Inception architecture.

• Is it that the 1x1 conv compresses the previous filter dimension to 1, before implementing the 5x5 conv ? Dec 13, 2016 at 7:51
• @LeonardLoo each 1x1 kernel reduces filter dimension to 1, but you can have multiple kernels in one 1x1 convolution, so the number of "filters" can be arbitrary of you choice. May 19, 2017 at 8:58
• It is clearly shown in the cited text: This leads to the second idea of the proposed architecture.... By ignoring the first paragraph of the cited paper The main idea of the Inception architecture is ..., this answer provides a partial explanation. In summary, the first reason, as explained in Network In Network and Xception: Deep Learning with Depthwise Separable Convolutions, is that the typical Inception module first looks at cross-channel correlations via a set of 1x1 convolutions.
– Liw
Jan 7, 2020 at 19:45
• Assuming you consider the shape of the output of the conv layer is (N,F,H,W) I think you speak about gray-scale images or just one channel image. A I true ? Nov 11, 2021 at 12:49

A 1x1 convolution simply maps an input pixel with all it's channels to an output pixel, not looking at anything around itself. It is often used to reduce the number of depth channels, since it is often very slow to multiply volumes with extremely large depths.

input (256 depth) -> 1x1 convolution (64 depth) -> 4x4 convolution (256 depth)

input (256 depth) -> 4x4 convolution (256 depth)

The bottom one is about ~3.7x slower.

Theoretically the neural network can 'choose' which input 'colors' to look at using this, instead of brute force multiplying everything.

• I would say that 1x1 maps not just one pixel to an output pixel, but it collapses all input pixel channels to one pixel. In your example in the first line, there are 256 channels for input, and each of the 64 1x1 kernels collapses all 256 input channels to just one "pixel" (real number). The result is that you have 64 channels now instead of 256 with the same spacial dimension, which makes 4x4 convolution computationally cheaper than in your second line example. May 19, 2017 at 9:03
• Good point, will update the post :) May 29, 2017 at 4:19
• Where did you get 3.7x slower from? May 22, 2018 at 1:31
• @CMCDragonkai Assume that the computation cost is proportional to the number of weights. In the later case, the num_weights = in_depth x out_depth x kernel_size = 256x256x4x4. In the first case, num_weights = 256x64x1x1+64x265x4x4=256x64x17. Take ratio and that gives 256/17~3.7. Check this paper page 9 for a visualisation on how num weights is related to input depth and output depth. arxiv.org/pdf/1603.07285.pdf Nov 5, 2018 at 14:20

The main reason I didn't understand 1x1 convolutions is because I didn't understand how $$any$$ convolutions really worked—the key factor is how computing a convolution of multiple channels/filters works. To understand this, I found this answer useful as well:

https://datascience.stackexchange.com/questions/9175/how-do-subsequent-convolution-layers-work

In particular, Type 2.2 is the correct description of a convolution there.

https://ai.stackexchange.com/questions/5769/in-a-cnn-does-each-new-filter-have-different-weights-for-each-input-channel-or

This answer explains how you have a separate filter for each in/out channel combination. After calculating each of these, the results get summed over the input channel axis leaving with output channel number of values.

Here's a video I found which helped me understand how a 1x1 convolution works.

https://www.coursera.org/lecture/convolutional-neural-networks/networks-in-networks-and-1x1-convolutions-ZTb8x

Here are the main things I got out of it:

• The input to a 1x1 convolution is usually previous convolutions which have size $$m$$ x $$n$$. But if there were $$f_1$$ filters in the last layer of convolutions, you're getting a $$(m, n, f_1)$$ shaped matrix. A 1x1 convolution is actually a vector of size $$f_1$$ which convolves across the whole image, creating one $$m$$ x $$n$$ output filter. If you have $$f_2$$ 1x1 convolutions, then the output of all of the 1x1 convolutions is size $$(m, n, f_2)$$.
• So a 1x1 convolution, assuming $$f_2 < f_1$$, can be seen as rerepresenting $$f_1$$ filters via $$f_2$$ filters. It lets the network train how to reduce the dimension most efficiently.
• Ok, this is the only place so far which properly explained the 1x1 convolution is actually a 'dot' product with $(m,n,f_1)$ May 21, 2019 at 16:42

## I will try to explain more intuitively and in-short with illustrations!

A 1*1 conv (a.k.a Network-in-Network)?

let's say you input is ($$n_H$$,$$n_W$$, $$n_{c_{prev}}$$). You can think of (1*1*$$n_{c_{prev}}$$) as a single neuron(a Fully Connected network-i.e why Network-in-Network) that's taking in ($$n_{c_{prev}}$$) numbers in the input multiplying them with (1*1*$$n_{c_{prev}}$$) and then adding and then taking ReLu & output ($$n_H$$, $$n_W$$) and if you have multiple filters($$n_c$$) then output would be ($$n_H$$, $$n_W$$, $$n_C$$).

So you can use pooling layer to reduce spacial dimensions($$n_H$$,$$n_W$$) and can use 1*1 conv to reduce the $$n_{c_{prev}}$$(i.e the number of channels) which saves a lot of computations. For example

Therefore,a take away

• You can use a 1x1 convolutional layer to reduce $$n_C$$ but not $$n_H$$, $$n_W$$.

• You can use a pooling layer to reduce $$n_H$$, $$n_W$$, and $$n_C$$.

In other words,

• what you are doing using 1*1 CONV filter is? you basically take "1*1*num_input_channel_of_featureMap" size weights & convolve(elementwise * followed by +) this volume over image/featureMap of size "WHnum_input_channel_of_featureMap" and what you get is an output of size "W*H".

• Now, you can use "#filters" number of such kind "1*1*num_input_channel_of_featureMap" and get the volume "WH#filter" as the final output!

• More, precisely, you are multiplying "1*1" such 32 different weights (1*1 Volume) with one slice at a same position in the input feature map followed by ReLu & get a single corresponding number generated, seen as green color pixel!

• 1*1 CONV helps to shrink the number of channels & save on computations in some networks(INCEPTIONS).!

• But, of course, if you wanna keep the number of channels to same as of input feature map, that's fine too, the one thing 1*1 CONV does is simple apply ReLU Non-linearity to it which allows it to learn complex functions in NN.

• A pooling layer only reduces nH and nW, and not nC. In your takeaway, you mention it wrong. Nov 11, 2020 at 14:28

In Machine Learning terminology, data often has more dimensions than is typically described e.g. 2d image data is normally actually 3d, with dimensions:

• $$w$$ the width of the image
• $$h$$ the height of the image
• $$k$$ (of size $$3$$) the RGB channels of the image

(in a greyscale image $$|k|=1$$).

When describing $$1\times1$$ convolutional layers, this usually implicitly means $$1\times1\times k$$, where $$k$$ is the number of channels - i.e. the filter reduces dimensionality across channels (e.g. averaging the value across the three separate RGB channels).

One more idea about dimensionality reduction in the context of 1x1 filters:

Take for example an 4096x8x8 fc7 layer from FCN. What happens if the next layer (call it fc8) is 2048x8x8 with filter size 1? fc7 is very deep inside the network, each of its 4096 features is semantically rich, but each neuron (e.g. input image is 250x250x3) has a large receptive field. In other words, if a neuron is very active, we know that somewhere in its semantic field there's a corresponding feature present.

Take for example a left-uppermost neuron in fc8 with a 1x1 filter. It connects to all 4096 neurons/features only in the same receptive field (upper-left corner of the image), each of which is activated by a single feature. Some (let's same 500) of them are very active. If the resulting neuron is also very active, it means it probably learnt to identify 1 or more features in this receptive field. After you've done this 2048 times for left-uppermost neurons in fc8, quite a few of them (e.g. 250) will be very active, meaning they 'collected' features from the same receptive field through fc7, and many very likely more than one.

If you keep reducing the dimensionality, a decreasing number of neurons will be learning an increasing number of features from the same receptive field. And since spatial parameters 8x8 remain the same, we do not change the 'view' of each neuron, thus do not decrease the spatial coarseness.

You may want to have a look at 'Fully Convolutional Networks' by Long, Shelhamer and Darrel.

• I think that this is another good intuition of the mechanism, but I'm going to have to come back to it, because I find the notation for the dimensions confusing, and especially "After you've done this 2048 times for left-uppermost neurons in fc8". Aug 15, 2018 at 3:13
• I didn't understand very well what you are trying to say, could you please make it clearer? Jan 15, 2019 at 22:24
• It is a good explanation...except for the dimensions which are re-arranged. Here we have many 1*1 filters - 2048 of them to be precise. Each 1*1 is like a fully connected single neuron. It connects to all 4096 values in fc7 and after the weighting, sum & activation ends up with 1 value. Maybe this neuron lights up when a certain pattern is observed in the 4096 values (which in turn light up when certain pattern is observed in the top left corner of the image). But of course this is not enough. We can’t reduce 4096 features to just 1. So we repeat this process…in this e.g we do it 2048 times. Jun 19, 2021 at 15:48
• And in all likelihood we would have learnt many more features in the process…not just 1. All this is for the receptive field for the upper left corner of the original image…which is why he says - “If you keep reducing the dimensionality, a decreasing number of neurons will be learning an increasing number of features from the same receptive field”. I am not sure if the word ‘increasing’ should be inserted in the above sentence though? Maybe the activation helps? Jun 19, 2021 at 15:49

The mathematical operation of convolution means to compute the product of two (continuous or discrete) functions over all possible shift-positions.

In a 2-dimensional (gray-level) image, a convolution is performed by a sliding-window operation, where the window (the 2-d convolution kernel) is a $v \times v$ matrix.

Image-processing applications of neural networks - including convolutional neural networks - have been reviewed in: [M. Egmont-Petersen, D. de Ridder, H. Handels. Image processing with neural networks - a review, Pattern Recognition, Vol. 35, No. 10, pp. 2279-2301, 2002].

## 3x3 vs 1x1 Convolution

import torch
import torch.nn as nn

image = torch.randn(1, 3, 1280, 1920)

# 3x3 convolution + padding, which keeps the spatial dimensions constant
model = nn.Conv2d(in_channels=3, out_channels=2, kernel_size=3, padding=1)
output = model(image)

print(output.shape)  # torch.Size([1, 2, 1280, 1920])

num_elements = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(num_elements)  # 56 == weights + bias

# weights = in_channels * out_channels * kernel_size * kernel_size
#    => 3 * 2 * 3 * 3 = 54
# bias = out_channels => 2

###

# 1x1 convolution
model = nn.Conv2d(in_channels=3, out_channels=2, kernel_size=1, padding=0)
output = model(image)

print(output.shape)  # torch.Size([1, 2, 1280, 1920])

num_elements = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(num_elements)  # 8 == weights + bias

# weights = in_channels * out_channels * kernel_size * kernel_size
#    => 3 * 2 * 1 * 1 = 6
# bias = out_channels => 2

# 8 << 56 :)