# representation of a convolutional layer as a fully connected one (matrix representation)

I'm surprised this isn't a duplicate, but Google seems to confirm that this is indeed the case.

What is the representation of a convolutional layer as a fully connected layer? A convolutional layer is nothing else than a discrete convolution, thus it must be representable as a matrix$$\times$$vector product, where the matrix is sparse with some well-defined, cyclic structure. However, what are neurons in this case? In the usual FC layer representation, each neuron has a vector input and a scalar output. But $$\mathbf{M}\cdot\mathbf{v}$$ is a vector, not a scalar. I must be missing something obvious.

PS of course $$\mathbf{M}$$ is not the input image - if the input is RGB, $$\mathbf{M}$$ must be some matricial representation of three matrices (R, G and B). If instead we're talking about a hidden layer, then $$\mathbf{M}$$ is a representation as a matrix of dozens or hundreds of matrices (channels).

• "In the usual FC layer representation, each layer has a vector input and a scalar output." -- isn't the output of an FC layer a vector? – shimao Jun 4 '19 at 21:55
• @shimao yes, of course - I put the blame on autocompletion! Fixed. – DeltaIV Jun 5 '19 at 16:00

For the 1D, 1-channel case, you may be interested in a related question and answer here.

In the 2D case*, if we flatten the input to the convolution $$x \in \mathbb{R}^{C\times H\times W}$$ into a vector $$x' \in \mathbb{R}^{CHW}$$ in the usual manner (such that $$x'_{iHW+jW+k} = x_{i,j,k}$$), and we have a convolutional kernel $$K \in \mathbb{R}^{D\times C\times P\times Q}$$ ($$D$$ is the out dimension and each filter has receptive field $$P$$ by $$Q$$) then we can define a weight matrix $$M \in \mathbb{R}^{DH'W' \times CHW}$$ ($$H'=H-P+1, W'=W-Q+1$$)such that the flattened version of $$y = \text{conv}(K,x)$$ can be written as $$y' = Mx'$$ as follows:

$$M_{s,t} = \begin{cases} K_{s,i,v-j,w-k} &\text{if } 0 \leq v-j < P \text{ and } 0 \leq w-k < Q\\ 0 &\text{otherwise } \end{cases}$$

Where $$i,j,k$$ are defined by $$t = iHW+jW+k$$ and $$j, $$k. and $$u,v,w$$ are defined by $$s = uH'W'+vW'+w$$ and $$v < H'$$ and $$w < W'$$.

You can see in each row of $$M$$, corresponding to a single entry in the output feature map, the only nonzero entries of that row are in the columns corresponding to the appropriate input receptive field.

*well I'm not masochistic enough to deal with strides, dilation, padding, separable filters, etc in this answer.

However, what are neurons in this case?

A neuron in a convolutional network (although I think it's usually not useful to think in terms of neurons), is a single entry in a feature (which is a vector) in a feature map (which is a 2D grid of features -- a 3D tensor).

Ok I agree the indexing notation is rather dense, here I'll write out an explicit example:

Our input $$x$$ is 1 by 3 by 3:

[
[
[1 2 3]
[4 5 6]
[7 8 9]
]
]


Each single value such as "1" or "5" here is a neuron.

We flatten this into the vector $$x'$$:

[1 2 3|4 5 6|7 8 9]


(to keep things sane, here and later, i use | to delimit every 3 elements, so you can see how they map onto the 1x3x3 input)

Meanwhile our kernel $$K$$ is 2 by 1 by 2 by 2:

[
[
[
[a b]
[c d]
]
]
[
[
[e f]
[g h]
]
]
]


We arrange this into the matrix $$M$$, which is 8 by 9:

[
[a b 0|c d 0|0 0 0]
[0 a b|0 c d|0 0 0]
[0 0 0|a b 0|c d 0]
[0 0 0|0 a b|0 c d]
[e f 0|g h 0|0 0 0]
[0 e f|0 g h|0 0 0]
[0 0 0|e f 0|g h 0]
[0 0 0|0 e f|0 g h]
]


Then $$Mx' = y'$$ computes

[1a+2b+4c+5d, 2a+3b+5c+6d, 4a+5b+7c+8d, 5a+6b+8c+9d, 1e+2f+4g+5h, 2e+3f+5g+6h, 4e+5f+7g+8h, 5e+6f+8g+9h]


Again as before, each scalar value here such as "1a+2b+4c+5d" is a single neuron.

We reshape this to 2 by 2 by 2 to recover $$y$$:

[
[
[1a+2b+4c+5d, 2a+3b+5c+6d]
[4a+5b+7c+8d, 5a+6b+8c+9d]
]
[
[1e+2f+4g+5h, 2e+3f+5g+6h]
[4e+5f+7g+8h, 5e+6f+8g+9h]
]
]


And you can see by inspection this is what we'd get by sliding the filter $$K$$ over the original $$x$$.

• "I think it's usually not useful to think in terms of neurons" - it's actually indispensable if you want to understand the literature on visualization CNN hidden layers, because the term is routinely used in that literature, pheraps unfortunately. Your answer is not clear to me - what is a neuron (or if you prefer, what are its inputs and output) in a CNN? In your notation, is the input $x'$ and the output $m_i\cdot x'$, i.e., one component of $y'$? Or to be more precise, is the neuron output $ReLU(y_i')=ReLU(m_i\cdot x')$? – DeltaIV Jun 6 '19 at 11:39
• it's a bit difficult (for me) to follow your notation. As an example, can you please write explicitly one or two (consecutive) rows of $M$? A simple case is fine - for example, $K \in \mathbb{R}^{3\times 1\times 3\times 3}$, and each of the 3 kernels is actually the same, for example, a sharpen kernel $[[0, -1, 0];[-1, 5, -1];[0, -1, 0]]$ – DeltaIV Jun 6 '19 at 11:47
• Ps I’m 💯 % with you about not dealing with strides, padding, etc. the basic 2D case is fine. – DeltaIV Jun 6 '19 at 16:25
• @DeltaIV I wrote out an explicit example -- hope it helps – shimao Jun 9 '19 at 5:44
• yeah, it's not. – shimao Jun 9 '19 at 7:53