Is building deep learning architectures a trial and error scheme?

Question

I have been reading many deep learning papers where each of them follow different architecture. I cannot see what the logical sense or the intuitive sense behind each layer in each architecture. I got a sense that many of those architectures are just arbitrary ones that we found them work in a perfect way for our application. I don't think science should be done like this. So I am sure I am missing somethings. Can someone point out for me to some general concepts and ideas I should follow to understand how I should design my deep learning architecture for my application? If there are materials/books or whatever help me to have a sense of how deep learning works (I know math, but not the logic or intuitive behind that) I appreciate it.

Answer 1

At the current stage, the neural network architecture selection is driven much more by empirical results rater than solid mathematical theory. Moreover, the network architecture (depth, breadth, activation functions, connections) are not the only decisions you have to make; also the optimization algorithm and its parameters interplay tightly with these choices. The specific dataset and the chosen loss function also define the loss surface along which you are optimizing. Sometimes even the hardware presents a limitation (e.g. amount of available GPU memory). There is simply not an universal, theoretically founded answer.

Of course, there are some intuitions: For example, you know how convolutions work, so it is easy to imagine what kind of information they can extract. Actually, most of the papers introducing some architectural tweaks, such as Batch normalization, Stochastic pooling, etc., provide such intuitive hints. It is your job to consider which of those make sense in your scenario. Any machine learning method has its hyperparameters that you have to tune. In case of neural networks, architecture is simply a hyperparameter (albeit an obscure one).

Besides, there are plenty threads dealing with this topic:

• How to choose the number of hidden layers and nodes in a feedforward neural network?
• Rules for selecting convolutional neural network hyparameters
• How to decide neural network architecture?
• Is there a heuristic for determining the size of a fully connected layer at the end of a CNN?