4
$\begingroup$

I'm preparing for an exam in Computer Vision. I came across with the following question from one of the exams:

What is the number of parameters of a convolution layer in a neural network, when the input size is $100 \times100 \times128$ and the output $100 \times100 \times256$, the convolution size is $3 \times3$, with and without the bias?

I noticed that they really like to ask this type of questions, although not sure what does it teach. I understand the general idea of the CNN architecture but eveytime they ask me to count, I get confuse.

I solved the following similar question:

What is the number of parameters of a linear layer in a neural network) called both fully-connected and dense) when the vector size at input 128 and output 256 with and without the bias?

My suggested solution:

Without bias: $$\#\text{params}=|\text{input}|\cdot|\text{output}|=128\cdot256=32,768$$

With bias: $$\#\text{params}=|\text{input}|\cdot\left(|\text{output}|+1\right)=128\cdot(256+1)=128\cdot257=32,896$$

How to count the parameters in the convolution layer above? In which other layers this "count parameters" question can pop up? (Pooling, etc.).

Improve this question
$\endgroup$
2
  • $\begingroup$ Please remember to add the self-study tag for this. $\endgroup$
    – Arya McCarthy
    Commented Apr 3, 2022 at 21:47
  • $\begingroup$ You might get some use out of drawings of mine. $\endgroup$
    – Dave
    Commented Apr 3, 2022 at 23:30

1 Answer 1

Reset to default
1
$\begingroup$

In a CNN layer, the number of parameters is determined by the kernel size and the number of kernels. The size of the input and output in the dimensions being convolved do not affect the number of parameters. So, in a standard 2-D CNN layer with 3-D input/output, only the 3rd dimension (often referred to as the channels) of the input/output matters. The number of input channels determines the 3rd dimension size of your kernels and the number of output channels is the number of kernels. The number of parameters in each kernel is simply the specified kernel size times the number of input channels (then $+1$ for the bias if using it), then multiply this by the number of kernels to get the total number of parameters.

In this case, your convolutional kernels are $3\times3\times128$ and you have $256$ of them, so:

Without bias: $$\begin{aligned}\#params&=kernel\_size\times|input\_channels|\times|output\_channels| \\&=3\times3\times128\times256 \\&=294912\end{aligned}$$

With bias: $$\begin{aligned}\#params&=(kernel\_size\times|input\_channels|+1)\times|output\_channels|\\&=(3\times3\times128+1)\times256\\&=295168\end{aligned}$$

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you! Are you sure that I need to ass the bias in the input channels? In other places I saw that they add it to the output channels so the formula is actually: $\#\text{params}=\text{kernel_size}\cdot|\text{input_channels}|\cdot\left(|\text{output_channels}|+\text{is_bias}\right)$. $\endgroup$
    – vesii
    Commented Apr 4, 2022 at 12:09
  • 1
    $\begingroup$ @vesii - Yes, each kernel has a bias, therefore there is one bias parameter for each output channel. Which is what you get with the formula I gave you. $\endgroup$
    – Lynn
    Commented Apr 4, 2022 at 23:06
  • 1
    $\begingroup$ An example from one of my models (its a 1-D CNN layer, rather than 2-D), it has 7 input channels, 32 output channels and kernel size is 5. From the tensorflow $summary()$ output it has 1152 parameters, $1152=(5\times7+1)\times\ 32$. Using your formula gives $5\times7\times(32+1)=1155$. $\endgroup$
    – Lynn
    Commented Apr 4, 2022 at 23:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.