# How does Krizhevsky's '12 CNN get 253,440 neurons in the first layer?

In Alex Krizhevsky, et al. Imagenet classification with deep convolutional neural networks they enumerate the number of neurons in each layer (see diagram below).

The network’s input is 150,528-dimensional, and the number of neurons in the network’s remaining layers is given by 253,440–186,624–64,896–64,896–43,264– 4096–4096–1000. ### A 3D View

The number of neurons for all layers after the first is clear. One simple way to calculate the neurons is to simply multiply the three dimensions of that layer (planes X width X height):

• Layer 2: 27x27x128 * 2 = 186,624
• Layer 3: 13x13x192 * 2 = 64,896
• etc.

However, looking at the first layer:

• Layer 1: 55x55x48 * 2 = 290400

Notice that this is not 253,440 as specified in the paper!

### Calculate Output Size

The other way to calculate the output tensor of a convolution is:

If the input image is a 3D tensor nInputPlane x height x width, the output image size will be nOutputPlane x owidth x oheight where

owidth = (width - kW) / dW + 1

oheight = (height - kH) / dH + 1 .

The input image is:

• nInputPlane = 3
• height = 224
• width = 224

And the convolution layer is:

• nOutputPlane = 96
• kW = 11
• kH = 11
• dW = 4
• dW = 4

(e.g. kernel size 11, stride 4)

Plugging in those numbers we get:

 owidth = (224 - 11) / 4 + 1 = 54 oheight = (224 - 11) / 4 + 1 = 54 

So we're one short of the 55x55 dimensions we need to match the paper. They might be padding (but the cuda-convnet2 model explicitly sets the padding to 0)

If we take the 54-size dimensions we get 96x54x54 = 279,936 neurons - still too many.

So my question is this:

How do they get 253,440 neurons for the first convolutional layer? What am I missing?

• Have you ever solved this? Just to be pedantic with your calculations: owidth and oheight would actually be 54.25. I tried to figure this out, and as a first step divided the supposed 253440 neurons among 96 filters, which yields 2640 neurons per filter. This isn't a square number. So either we both have a misunderstanding here, or there might be a mistake by the authors... Have you contacted them? – anderas Feb 21 '15 at 22:25
• same with me, this is very confuse me. btw there is true the input is 224x224x3? i think it must be 227x227x3. let we see if we have 227x227, 5 cell on first left and 5 cell on last right cannot be the center of kernel convolution with size 11x11. So the first center of kernel is cell (6,6) and the last of center kernel in first row is cell(6x222). With stride-4 we will get the center of kernel on row-sixth are: cell on column :6,10,14, ...,222 and simple formulation for the center of kernel-k is on column = 6+(k-1)*4 so that column 222 is the k-th center = (222-6)/4 +1 = 55. – user89674 Sep 17 '15 at 7:09
• Note that 48*48*55*2=253440, so it's possible they had a typo when calculating the number of neurons in the first layer (multiplied by 48 instead of 55). – tsiki Nov 19 '15 at 17:52
• – kenorb Jun 27 '16 at 18:55
• @Firebug This is an interesting usage of the [references] tag. I thought we use it only for questions that ask for references. But perhaps I was wrong. Do you use it differently? – amoeba Sep 23 '16 at 19:50

## 2 Answers

From the stanfords note on NN:

Real-world example. The Krizhevsky et al. architecture that won the ImageNet challenge in 2012 accepted images of size [227x227x3]. On the first Convolutional Layer, it used neurons with receptive field size F=11, stride S=4 and no zero padding P=0. Since (227 - 11)/4 + 1 = 55, and since the Conv layer had a depth of K=96, the Conv layer output volume had size [55x55x96]. Each of the 55*55*96 neurons in this volume was connected to a region of size [11x11x3] in the input volume. Moreover, all 96 neurons in each depth column are connected to the same [11x11x3] region of the input, but of course with different weights. As a fun aside, if you read the actual paper it claims that the input images were 224x224, which is surely incorrect because (224 - 11)/4 + 1 is quite clearly not an integer. This has confused many people in the history of ConvNets and little is known about what happened. My own best guess is that Alex used zero-padding of 3 extra pixels that he does not mention in the paper.

These notes accompany the Stanford CS class CS231n: Convolutional Neural Networks for Visual Recognition. For questions/concerns/bug reports regarding contact Justin Johnson regarding the assignments, or contact Andrej Karpathy regarding the course notes

• Is it possible to give credit by naming the author of those notes? – Silverfish Oct 27 '16 at 21:25
• Authors are clearly self referenced in the notes, see cs231n.github.io – Bacon Oct 27 '16 at 22:16
• Although they are clearly referenced if you follow the link, the quote appears here too, so there should be credit given here also. That's partly courtesy, but also since the link may stop working (e.g. if the material is removed/paywalled/moved to a new address). Unfortunately we have a severe problem with "link rot" on this site - while we intend to serve as a long-term repository of high-quality answers to statistical questions, many of our older answers have been rendered obsolete by links that no longer work. So it's generally safer to try to make answers as self-contained as possible. – Silverfish Oct 27 '16 at 23:35

This paper is really confusing. First off, the input size of images is incorrect 224x224 does not give an output of 55. Those neurons are simply just like grouped pixels in one, so the output is a 2D image of random values (neuron values). So basically the number of neurons = widthxheightxdepth, no secrets are there to figure this out.