5
$\begingroup$

VGG16 is a Convolutional Neural Network architecture, presented in the paper Very Deep Convolutional Networks for Large-Scale Image Recognition (ILSVRC-2014 conference presentation here). As explained in the paper,

The width of convolutional layers (the number of channels) is rather small, starting from 64 in the first layer and then increasing by a factor of 2 after each max-pooling layer, until it reaches 512.

Why is the number of channels doubled after each convolutional layer?

Jeremy Howard in the fast.ai course says it is not to lose information. Is it important to maintain the same number of weights on every layer for a convolutional neural network? Can it be beneficial to lose information?

$\endgroup$
  • $\begingroup$ Please add the link to the lecture in which Jeremy Howard says that. $\endgroup$ – DeltaIV Nov 9 '17 at 8:24
6
$\begingroup$

You should really ask in the course forum :) or contact Jeremy on Twitter, he's a great guy. Having said that, the idea is this: subsampling, aka pooling (max pooling, mean pooling, etc.: currently max pooling is the most common choice in CNNs) has three main advantages:

  1. it makes your net more robust to noise: if you alter slightly each neighborhood in your input layer, then the mean of each neighborhood won't change a lot (the smoothing effect of the sample mean). The max doesn't have this smoothing effect, however since it's the largest activation value, the relative variation due to noise is (on average) smaller than for other pixels.
  2. it introduces some level of translation invariance. By reducing the number of features in the output layer, if you move slightly the input image, chances are the output of the subsampling layer won't change, or it will change less. See here for a nice picture
  3. also, by reducing the number of features, the computational effort in training and predicting is reduced. Also overfitting becomes less likely.

However, not everyone agrees with point 3. In the famous Alexnet paper, which can be considered as the "rebirth" of CNNs, the authors used overlapping neighborhoods (i.e., strides along x and y smaller than the extension of the subsampling neighborhood along x and y respectively) in order to get the same number of features for the input and the output of the subsampling layer. This makes the model more flexible, which is what Jeremy was hinting at. You get a more flexible model, at the risk of more overfitting - but you can use other Deep Learning tools to fight overfitting. It's really a design choice - you'll typically need validation data sets to try different architectures and see what works best.

EDIT: It just occured to me that I have misunderstood what you were asking for. VGG16, unlike Alexnet, uses nonoverlapping max pooling (see chapter 2.1 of the paper I linked, right at the end of the first paragraph). Thus the size of the channels does reduce by 50% after each pooling layer. This is compensated by the doubling of the width of the convolutional layers: enter image description here

actually, it doesn't happen always - after the penultimate maxpool, the width remains 512, the same as before pooling. Again, this is a design choice: it's not set in stone, as confirmed by the fact that they don't follow this rule for the last convolutional layer. However, it's by far the most common design choice: for example, both LeNet and Alexnet follow this rule, even though LeNet uses nonoverlapping pooling (the size of each channel is halved, as for VGG16), while Alexnet uses overlapping pooling. The idea is simple - you introduce maxpooling to add robustness to noise and to help making the CNN translation equivariant, as I said before. However, you also don't want to throw away information contained in the image, together with the noise. To do that, for each convolutional layer you double the number of channels. This means that you have twice as many "high level features", so to speak, even if each of them contains half as many pixels. If your input image activates one of this high level features, their activation will be passed to following layers.

Granted, this added flexibility adds a risk of overfitting, which they combat with the usual techniques (see chapter 3.1): $L_2$ regularization and dropout for the last two layers, learning rate decay for the whole net.

$\endgroup$
1
$\begingroup$

If you think that first convolutional layer extracts simple features, like lines, than the next convolutional layer has to combine these lines into more complected structure. Basically, each channel in the next layer responsible for the combination of different lines into more complicated structure. Since structure getting more complicated there are getting more features that you can construct from simple lines.

If you think of lego-building classifier. First layer learns basic building lego-blocks, the second one looks for what things you can construct using these blocks and clearly there are way more things that you can build from basic building blocks.

Another analogy is a digital number classifier. You have 7 LEDs in one matrix, which means that you can learn using first layer all 7 lines (or maybe 2 if you learn that there is one horizontal and one vertical line, just in different places).

enter image description here

Using just 7 features you can construct 10 different digits. In addition to it, you can construct letters, like L, A, H and so on. So in the next convolutional layer, you should have more than 7 layers in order to be able to learn different letters and digits.

These are toy-examples, but the same intuition you can extend to more realistic problems that large convolutional neural networks are trying to solve

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.