How to think about the architecture of the Convolutional Neural Network?

Question

Recently, I've started to learn more about CNNs to use them in some computer vision tasks.

At the moment, I have roughly good knowledge about different parts of a CNN such as layers, solvers, loss functions, forward and backward passes, initialization methods and hyper parameters.

But, there is still a question in my mind that I do not have a suitable answer for it.

Whenever I read a paper, I notice that authors are trying to use their own CNN architecture to do some specific tasks. They put some layers together to generate the intended output. For example, it is very common to have multiple convolutional layers to obtain a hierarchy of image features in different scales and abstraction levels.

But in many cases, they put some fully connected layers successively at the very end of the network to feed the last classifier (e.g. softmax layer).

What is the role of these multiple FC layers exactly and how should I think about them and their quantity? Is this also another hyper parameter?

In general, how should we start thinking about the architecture of the CNN in a specific task such as let's say image segmentation?

Thanks for all the attentions.

Answer 1

Intuitively - the final level processes data representing some higher-level abstractions e.g. a cow, grass meadow background

So a "fully connected" level is needed for all of these to affect the final output.

A lower level might handle spatially related features e.g. 4 legs near each other, so it is "not interested" in other parts of the picture.

Etc

Answer 2

The final layers in a convolutional neural network could be seen as sorting neurons. The initial convolutional layers have 'picked out' features form the image, in the first convolutional layer, and combinations of those features (in the second convolutional layer). This is great but the layers haven't decided anything about what class (category) these features belong to. The final layers take the features that the convolutional layers have identified and piece them together to give a conclusion as to what class they are representing. The final layers 'make sense' of what the convolutional layers are 'seeing'.

Answer 3

Hopefully someone will come in with a more mathematically rigorous answer, but in a lot of cases it seems to just be about cramming your data into the shape you need.

For example, in Magenta Coconet, the algorithm is trying to inpaint a musical score, and so the output dimensions are the same as the input dimensions (with a small exception that isn't relevant to your question), so they don't even use fully connected layers at the end. They just use as many hidden layers as sounded good to them, and then threw on input and output layers that made the sizes right.

In a classification problem though, you have to change higher dimensional data (images or something) into a 1 dimensional array (categories), and so fully connected layers can be used to reshape the data.

Answer 4

Fully connected layers are mathematically equivalent to 1x1 Convolutional layers, so you can still apply your previous intuition here.