17
$\begingroup$

When using a fully-connected network (FCN), I have problem understanding how fully-connected (FC) layer to convolutional layer conversion actually works, even after reading http://cs231n.github.io/convolutional-networks/#convert.

In their explanation, it's said that: enter image description here

In this example, as far as I understood, the converted CONV layer should have the shape (7,7,512), meaning (width, height, feature dimension). And we have 4096 filters. And the output of each filter's spatial size can be calculated as (7-7+0)/1 + 1 = 1. Therefore we have a 1x1x4096 vector as output.

Although the converted layer can give us output with same size, how can we make sure they are indeed functionally equivalent? It's mentioned in the later part of the post that we need to reshape the weight matrix of FC layer to CONV layer filters. But I am still confusing about how to actually implement it.

Any explanation or link to other learning resource would be welcome.

$\endgroup$
0

2 Answers 2

8
$\begingroup$

Inspired by @dk14 's answer, now I have a clearer mind on this question, though I don't completely agree with his answer. And I hope to post mine online for more confirmation.

On a vanilla case, where the input of original AlexNet is still (224,224,3), after a series of Conv layer and pooling, we reach the last Conv layer. At this moment, the size of the image turns into (7,7,512).

At the converted Conv layer(converted from FC1), we have 4096 * (7,7,512) filters overall, which generates (1,1,4096) vector for us. At the second converted Conv layer(converted from FC2), we have 4096 * (1,1,4096) filters, and they give us a output vector (1,1,4096). It's very important for us to remember that, in the conversion, filter size must match the input volume size. That's why we have one by one filter here. Similarily, the last converted Conv layer have 1000 * (1,1,4096) filters and will give us a result for 1000 classes.

The processed is summarized in the post: http://cs231n.github.io/convolutional-networks/#convert. enter image description here

In FC1, the original matrix size should be (7*7*512, 4096), meaning each one of the 4096 neuron in FC2 is connected with every neuron in FC1. While after conversion, the matrix size becomes (7,7,512,4096), meaning we have 4096 (7,7,512) matrixes. It's like taking out each row of the original gigantic matrix, and reshape it accordingly.

$\endgroup$
4
$\begingroup$

Let's start with $F = 7$, $P = 0$, $S = 1$ notion. What does it actually mean:

  • $F = 7$: receptive field size is set to a maximum value (7 for 1D, 7x7 for 2D) which implies no parameter sharing (as there is only one receptive field), which is default for MLP. If F was equal to 1, all connections (from the image above) would always have an identical weight.

  • $S = 1$: stride equals to 1, which means that no neurons on the next layer is going to be removed (see figure below). Given $F = 7$ if we had stride = 2, the number of next-layer nodes would be twice smaller. enter image description here Source: http://cs231n.github.io/convolutional-networks

  • $P = 0$: no zero padding, as we don't need it for a full receptive field (there is no uncovered units as you can see from image above).

Those three conditions basically guarantee that connectivity architecture is exactly same as for canonical MLP.


Attempt to answer your question about reshaping matrices:

Example of reshaping in Python's Numpy library: numpy.reshape

My guess is that the author meant that FCN usually has 1D output "vector" (from each layer) instead of 2D matrix. Let's say, the first layer of FC-network returns 1x1x4096 output matrix as it doesn't care about image's dimensions - it stacks all dimensions into one vector (put each rows on top of another). You can guess that next layer's weight matrix is gonna have corresponding shape (4096x4096) that combines all possible outputs). So when you convert it to a convolutional receptive field - you'll probably have to move your activations to 2D, so you need 64x64 activations and, I guess, something like 64x64x4096 tensor for receptive field's weights (since $S=1$).

The quote from the article that demonstrates "reshaping":

For example, if 224x224 image gives a volume of size [7x7x512] - i.e. a reduction by 32, then forwarding an image of size 384x384 through the converted architecture would give the equivalent volume in size [12x12x512], since 384/32 = 12. Following through with the next 3 CONV layers that we just converted from FC layers would now give the final volume of size [6x6x1000], since (12 - 7)/1 + 1 = 6. Note that instead of a single vector of class scores of size [1x1x1000], we’re now getting and entire 6x6 array of class scores across the 384x384 image

Example (for activations of some layer):

1
2
3
4

|
|
\/

1 3
2 4

In order to show weights reshaping (to fit 2D image), I'd have to draw square into cube conversion. However, there is some demos on the internet:

enter image description here Source: http://nuit-blanche.blogspot.com/2016/09/low-rank-tensor-networks-for.html

P.S. However, I have some confusion about AlexNet example: it seems like mentioned $F=1$ just means "full" parameter sharing across non-existent dimensions (1x1). Otherwise, it won't be completely equivalent to an MLP with no parameter sharing - but maybe that's what was implied (scaling small FC-network into a large CNN).


So, what's the point?

to “slide” the original ConvNet very efficiently across many spatial positions in a larger image

Basically it allows you to scale a FC-network trained on small portions/images into a larger CNN. So in that case only small window of resulting CNN will be initially equivalent to an original FCN. This approach gives you ability to share parameters (learned from small networks) across large networks in order to save computational resources and apply some kind of regularization (by managing network's capacity).


Edit1 in response to your comment.

Example of $N = 5$ (sorry I was lazy to draw 7 neurons), $F=5$, $S=2$ :

enter image description here

So you can see that S = 2 can be applied even for receptive field with maximum size, so striding can be applied without parameter sharing as all it does is just removing neurons.

And parameter sharing strategies could be different. For instance, you can't tell about my last figure wether parameter are shared between neurons or not.

$\endgroup$
9
  • $\begingroup$ Thanks for your comments! But I have a different understanding on S, P, F though. Starting with F: As you indicated, it's set to 7 to ensure no parameter sharing, but I think it also indicates full connectivity between layers when filter size equals to input spatial size and no zero padding. P: no zero padding. This is extremely important because if it's more than zero, each next layer's neuron is connected to previous neurons at least twice, while FC layer's neurons are not.(Only once) S: In the FC -> CONV conversion, I think it doesn't matter what S is. Because there's no sliding at all. $\endgroup$
    – Sta_Doc
    Commented Feb 23, 2017 at 1:25
  • $\begingroup$ @Bato Oh you're right about P, corrected it. About F and S: Basically only F = 7 matters, but as I mentioned S = 2 reduces size of the next layer twice (imagine it not as a sliding but a stride that removes every second neuron), so it should be 1. " each next layer's neuron is connected to previous neurons at least twice" - I don't understand it. Zero padding is needed only when F < 7 coz "edge input neurons" have less connections then (so they less "power"). $\endgroup$
    – dk14
    Commented Feb 23, 2017 at 3:18
  • $\begingroup$ you're right. The 'S' doesn't matter only when F=7 or the input size remains unchanged, and I'm not sure whether it can be values other than one. I feel like even if S=2, we can still find its corresponding Conv layer. Say, in the case where input size is (7,7,512), we use F=7,S=5,P=0,K=4096 to reconstruct the Conv layer. When the input size changes from (224,224) to (384,384), we will have a (2,2,1000) array at the output. The difference here is, when using S=1, we are iterating the original ConvNet over 36 locations in the larger image. But when S=5, we just iterate 4 locations. $\endgroup$
    – Sta_Doc
    Commented Feb 23, 2017 at 4:35
  • $\begingroup$ by saying "each next layer's neuron is connected to previous neurons at least twice" I mean there should be no sliding or jumping of the filter. Because in fully connected layer, each neuron in the next layer will just have one matrix multiplication with the previous neurons. If the filter is sliding or jumping, it's equivalent to two matrix multiplications in one neuron in FC layer, which is not correct. And indeed setting F = input size and P=0 can ensure it. That's why I feel S is not that important in this case @dk14 $\endgroup$
    – Sta_Doc
    Commented Feb 23, 2017 at 4:42
  • $\begingroup$ @Bato. I think our interpretations of S are a little different. I see S as a stride not in the meaning of "sliding step", but in the meaning "overstep". If you have just one layer "x x x x x x x" of neurons, applying stride = 2 to it would result "x _ x _ x _ x", stride = 3 "x _ _ x _ _ x", so even if F = 7 - you gonna loose neurons in the next layer if you apply $S > 1$, so it would be "fully-connected" only for those x-neurons, _-neurons will loose all their connections. $\endgroup$
    – dk14
    Commented Feb 23, 2017 at 5:05

Not the answer you're looking for? Browse other questions tagged or ask your own question.