Spatial arrangement. We have explained the connectivity of each neuron in the Conv Layer to the input volume, but we haven't yet
discussed how many neurons there are in the output volume or how they
are arranged. Three hyperparameters control the size of the output
volume: the depth, stride and zero-padding. We discuss these
- First, the depth of the output volume is a hyperparameter: it corresponds to the number of filters we would like to use, each
learning to look for something different in the input. For example, if
the first Convolutional Layer takes as input the raw image, then
different neurons along the depth dimension may activate in presence
of various oriented edged, or blobs of color. We will refer to a set
of neurons that are all looking at the same region of the input as a
depth column (some people also prefer the term fibre).
- Second, we must specify the stride with which we slide the filter. When the stride is 1 then we move the filters one pixel at a
time. When the stride is 2 (or uncommonly 3 or more, though this is
rare in practice) then the filters jump 2 pixels at a time as we slide
them around. This will produce smaller output volumes spatially.
- As we will soon see, sometimes it will be convenient to pad the input volume with zeros around the border. The size of this
zero-padding is a hyperparameter. The nice feature of zero padding is that it will allow us to control the spatial size of the output
volumes (most commonly as we'll see soon we will use it to exactly
preserve the spatial size of the input volume so the input and output
width and height are the same).
We can compute the spatial size of the output volume as a function of
the input volume size ($W$), the receptive field size of the Conv
Layer neurons ($F$), the stride with which they are applied ($S$), and
the amount of zero padding used ($P$) on the border. You can convince
yourself that the correct formula for calculating how many neurons
"fit" is given by $(W - F + 2P)/S + 1$. For example for a 7x7 input
and a 3x3 filter with stride 1 and pad 0 we would get a 5x5 output.
With stride 2 we would get a 3x3 output. Lets also see one more
Illustration of spatial arrangement. In this example there is only one
spatial dimension (x-axis), one neuron with a receptive field size of
F = 3, the input size is W = 5, and there is zero padding of P = 1.
Left: The neuron strided across the input in stride of S = 1,
giving output of size (5 - 3 + 2)/1+1 = 5. Right: The neuron
uses stride of S = 2, giving output of size (5 - 3 + 2)/2+1 = 3.
Notice that stride S = 3 could not be used since it wouldn't fit
neatly across the volume. In terms of the equation, this can be
determined since (5 - 3 + 2) = 4 is not divisible by 3.
The neuron weights are in this example [1,0,-1] (shown on very right),
and its bias is zero. These weights are shared across all yellow
neurons (see parameter sharing below).
Use of zero-padding. In the example above on left, note that the input dimension was 5 and the output dimension was equal: also 5. This
worked out so because our receptive fields were 3 and we used zero
padding of 1. If there was no zero-padding used, then the output
volume would have had spatial dimension of only 3, because that it is
how many neurons would have "fit" across the original input. In
general, setting zero padding to be $P = (F - 1)/2$ when the stride is
$S = 1$ ensures that the input volume and output volume will have the
same size spatially. It is very common to use zero-padding in this way
and we will discuss the full reasons when we talk more about ConvNet
Constraints on strides. Note again that the spatial arrangement hyperparameters have mutual constraints. For example, when the input
has size $W = 10$, no zero-padding is used $P = 0$, and the filter
size is $F = 3$, then it would be impossible to use stride $S = 2$,
since $(W - F + 2P)/S + 1 = (10 - 3 + 0) / 2 + 1 = 4.5$, i.e. not an
integer, indicating that the neurons don't "fit" neatly and
symmetrically across the input. Therefore, this setting of the
hyperparameters is considered to be invalid, and a ConvNet library
could throw an exception or zero pad the rest to make it fit, or crop
the input to make it fit, or something. As we will see in the ConvNet
architectures section, sizing the ConvNets appropriately so that all
the dimensions "work out" can be a real headache, which the use of
zero-padding and some design guidelines will significantly alleviate.
Real-world example. The Krizhevsky et al.
architecture that won the ImageNet challenge in 2012 accepted images
of size [227x227x3]. On the first Convolutional Layer, it used neurons
with receptive field size $F = 11$, stride $S = 4$ and no zero padding
$P = 0$. Since (227 - 11)/4 + 1 = 55, and since the Conv layer had a
depth of $K = 96$, the Conv layer output volume had size [55x55x96].
Each of the 55*55*96 neurons in this volume was connected to a
region of size [11x11x3] in the input volume. Moreover, all 96 neurons
in each depth column are connected to the same [11x11x3] region of the
input, but of course with different weights. As a fun aside, if you
read the actual paper it claims that the input images were 224x224,
which is surely incorrect because (224 - 11)/4 + 1 is quite clearly
not an integer. This has confused many people in the history of
ConvNets and little is known about what happened. My own best guess is
that Alex used zero-padding of 3 extra pixels that he does not mention
in the paper.
The MIT License (MIT)
Copyright (c) 2015 Andrej Karpathy
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.*
Often, in 1D CNN for natural language processing, the number of features is the number of filters, since a max pooling layer is commonly added after the convolution layer (this way it can easily be applied to input of variable size):