How to convert fully connected layer into convolutional layer? When using a fully-connected network (FCN), I have problem understanding how fully-connected (FC) layer to convolutional layer conversion actually works, even after reading http://cs231n.github.io/convolutional-networks/#convert. 
In their explanation, it's said that:

In this example, as far as I understood, the converted CONV layer should have the shape (7,7,512), meaning (width, height, feature dimension). And we have 4096 filters. And the output of each filter's spatial size can be calculated as (7-7+0)/1 + 1 = 1. Therefore we have a 1x1x4096 vector as output.  
Although the converted layer can give us output with same size, how can we make sure they are indeed functionally equivalent? It's mentioned in the later part of the post that we need to reshape the weight matrix of FC layer to CONV layer filters. But I am still confusing about how to actually implement it. 
Any explanation or link to other learning resource would be welcome.
 A: Inspired by @dk14 's answer, now I have a clearer mind on this question, though I don't completely agree with his answer. And I hope to post mine online for more confirmation. 
On a vanilla case, where the input of original AlexNet is still (224,224,3), after a series of Conv layer and pooling, we reach the last Conv layer. At this moment, the size of the image turns into (7,7,512).
At the converted Conv layer(converted from FC1), we have 4096 * (7,7,512) filters overall, which generates (1,1,4096) vector for us. At the second converted Conv layer(converted from FC2), we have 4096 * (1,1,4096) filters, and they give us a output vector (1,1,4096). It's very important for us to remember that, in the conversion, filter size must match the input volume size. That's why we have one by one filter here. Similarily, the last converted Conv layer have 1000 * (1,1,4096) filters and will give us a result for 1000 classes. 
The processed is summarized in the post:  http://cs231n.github.io/convolutional-networks/#convert.

In FC1, the original matrix size should be (7*7*512, 4096), meaning each one of the 4096 neuron in FC2 is connected with every neuron in FC1. While after conversion, the matrix size becomes (7,7,512,4096), meaning we have 4096 (7,7,512) matrixes. It's like taking out each row of the original gigantic matrix, and reshape it accordingly. 
A: Let's start with $F = 7$, $P = 0$, $S = 1$ notion. What does it actually mean:


*

*$F = 7$: receptive field size is set to a maximum value (7 for 1D, 7x7 for 2D) which implies no parameter sharing (as there is only one receptive field), which is default for MLP. If F was equal to 1, all connections (from the image above) would always have an identical weight. 

*$S = 1$: stride equals to 1, which means that no neurons on the next layer is going to be removed (see figure below). Given $F = 7$ if we had stride = 2, the number of next-layer nodes would be twice smaller.

Source: http://cs231n.github.io/convolutional-networks

*$P = 0$: no zero padding, as we don't need it for a full receptive field (there is no uncovered units as you can see from image above).
Those three conditions basically guarantee that connectivity architecture is exactly same as for canonical MLP.

Attempt to answer your question about reshaping matrices:
Example of reshaping in Python's Numpy library: numpy.reshape
My guess is that the author meant that FCN usually has 1D output "vector"  (from each layer) instead of 2D matrix.
Let's say, the first layer of FC-network returns  1x1x4096 output matrix as it doesn't care about image's dimensions - it stacks all dimensions into one vector (put each rows on top of another). You can guess that next layer's weight matrix is gonna have corresponding shape (4096x4096) that combines all possible outputs). So when you convert it to a convolutional receptive field - you'll probably have to move your activations to 2D, so you need 64x64 activations and, I guess, something like 64x64x4096 tensor for receptive field's weights (since $S=1$).
The quote from the article that demonstrates "reshaping":

For example, if 224x224 image gives a volume of size [7x7x512] - i.e. a reduction by 32, then forwarding an image of size 384x384 through the converted architecture would give the equivalent volume in size [12x12x512], since 384/32 = 12. Following through with the next 3 CONV layers that we just converted from FC layers would now give the final volume of size [6x6x1000], since (12 - 7)/1 + 1 = 6. Note that instead of a single vector of class scores of size [1x1x1000], we’re now getting and entire 6x6 array of class scores across the 384x384 image

Example (for activations of some layer):
1
2
3
4

|
|
\/

1 3
2 4

In order to show weights reshaping (to fit 2D image), I'd have to draw square into cube conversion. However, there is some demos on the internet:

Source: http://nuit-blanche.blogspot.com/2016/09/low-rank-tensor-networks-for.html
P.S. However, I have some confusion about AlexNet example: it seems like mentioned $F=1$ just means "full" parameter sharing across non-existent dimensions (1x1). Otherwise, it won't be completely equivalent to an MLP with no parameter sharing - but maybe that's what was implied (scaling small FC-network into a large CNN).

So, what's the point?

to “slide” the original ConvNet very efficiently across many spatial positions in a larger image

Basically it allows you to scale a FC-network trained on small portions/images into a larger CNN. So in that case only small window of resulting CNN will be initially equivalent to an original FCN. This approach gives you ability to share parameters (learned from small networks) across large networks in order to save computational resources and apply some kind of regularization (by managing network's capacity).

Edit1 in response to your comment.
Example of $N = 5$ (sorry I was lazy to draw 7 neurons), $F=5$, $S=2$ :

So you can see that S = 2 can be applied even for receptive field with maximum size, so striding can be applied without parameter sharing as all it does is just removing neurons. 
And parameter sharing strategies could be different. For instance, you can't tell about my last figure wether parameter are shared between neurons or not.
