2
$\begingroup$

If the input was a 28 x 28 x 192 image; and there are 192 1x1 convolutions (with RELUs) ie the filter layer is 1 x 1 x 192 x 192. This would produce an output of dimensions 28 x 28 x 192.

I understand that a benefit of the 1x1 convolution is the non-linearity element (ie the RELU).

Is there a reason why one would keep the number of channels unchanged (as per the example above) as opposed to just using a RELU function without the 1x1 convolution?

$\endgroup$
3
$\begingroup$

Consider just a single feature column for clarity, so if the output is $y$ and the input is $x$ then $y = \text{relu}(w^Tx+b)$.

This is useful in itself: Suppose $x_1$ is the roughly speaking, the amount of cat detected and $x_2$ is the amount of dog detected. Then $w^Tx$ can compute the amount of cat minus the amount of dog, and if $b = 0$, then one component of $y$ indicates whether there is more cat than dog.

This same operation can't be carried out without the linear function within the 1x1 conv layer.

$\endgroup$
3
$\begingroup$

No. Generally, number of filters in every layer is a hyperparameter that has different optimal value for every dataset. The same holds also for the size of the filters: Convolutions 1x1 are not so much different from convolutions 3x3; they cannot aggregate information from neighboring spatial locations, but they still can aggregate information over channels in every spatial location. As explained by @shimao, plain ReLU cannot do that.

The rule of thumb that is used nowadays is that when you reduce the resolution (by strided convolutions or pooling), you increase the number of channels (thus transforming local information to global). But this is also mainly justified by empirical success.

$\endgroup$
1
$\begingroup$

A 1 x 1 convolution serves to emphasize or ignore single pixels. Its purpose is to reduce dimensionality. I know it sounds contradictory. However, the pixels of the output have either a higher or lower value than before. The following convolution layers will learn faster which pixels are more important in the diagnosis.

EDIT: By reduce dimensionality, I meant discarding pixels which have no importance to the final classification, I did not mean reducing the size of the grid or output.

$\endgroup$
  • 2
    $\begingroup$ The reduction in dimensionality happens if you use less conv. filters than is the dimension of the input. It doesn't have anything to do with the magnitude of the signals. $\endgroup$ – Jan Kukacka May 3 '18 at 7:51
  • $\begingroup$ @JanKukacka thank you for pointing out that I am using the wrong Wording. I will edit my answer to fix it. $\endgroup$ – Amani May 4 '18 at 7:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.