Suppose that I have want to make a model that does something. Now when I search about the topic in Google or YouTube, I find many related tutorials and it seems like some clever programmer had already implemented that model with Deep learning.

But how do they know that what type of layers, what type of activation functions, loss functions, optimizer, number of units etc. they need to solve that certain problem using deep learning.

Are there any techniques for knowing this, or its just a matter of understanding and experience? Also it would be very helpful if somebody could point me to some videos or articles answering my question.


I have some good news and some bad news for you. The good news is that there are many problems for which we know which architecture works best, because of previous research. The bad news is that since today we don't have a good theory of generalization for Deep Networks, we lack theoretical guidance about how to select an architecture for a new problem (however, read here for some insights). Thus, in general the most honest answer is that "it's just a matter of understanding and experience". On the other hand, for some specific fields we can give more canned suggestions:

Computer Vision

We know that the Convnet family of architectures works very well for image classification: LeNet, Alexnet, VGGNet, ResNets, etc. You can train a beefed-up version of LeNet on a non-GPU laptop, and that's a great way to start learning about them. I suggest you start from this Keras implementation


and try to improve it a bit, by reproducing in Keras the section Implementing a CNN in the TensorFlow layers API of this Jupyter notebook. By the way, I can't recommend the second edition of Sebastian Raschka's book highly enough - it's a great way to gain practical knowledge about Machine Learning and Deep Learning. Instead of wasting time reading multiple tutorials on the Internet, get the book - you'll get a more solid understanding of the subject, also because quite a few of the most cited blog posts on Convolutional Neural Networks are basically summaries of the first edition of the book.

If you want to train architectures which will perform well on realistic, big data sets (such as CIFAR-100 or ImageNet), you need to have access to a GPU cluster.

Natural Language Processing

Here we know that RNNs work well. Actually, the "simple" RNN architecture known as LSTM delivers much better results than most people would commonly expect, as shown in this paper: On the State of the Art of Evaluation in Neural Language Models. The paper also highlights a big limit of modern Deep Learning research: a lot of papers don't care enough about repeatability and reproducibility of results, and some results than are presented as the new state of the art, are instead just due to uncontrolled experimental variation. Again, Raschka's book can be quite useful to start learning about RNNs, together with the corresponding Jupyter notebook.

The general case

If you want to tackle a known problem, but maybe with a new data set (for example, you want to perform image classification of car parts because you work for a car manufacturer), you need to use model selection techniques, such as for example cross-validation. You build different networks (different number of layers, different activation functions, etc.) and choose the one with the smallest cross-validation error. Then, you retrain it on the full data set, and you use it for prediction.

However, since the number of alternatives can be prohibitive, you can use some automated machine learning frameworks which help you explore the space of possible networks, such as for example:

If you need to work on big data sets, these tools won't work (they are based on scikit-learn, so there's no support for GPUs, currently). You may have a look at this paper, Large-Scale Evolution of Image Classifiers: like the other one I linked, this one takes proper care to ensure repeatability of results.

If you want to attack a new problem (say, Neural Program Synthesis) for which we still have no idea of which architectures work best, probably your best bet is to attend NIPS and ICML (or stalk the right sections of arXiv), in the hope that someone has already tackled your problem.


If you want to use Deep Learning, you must know what is it good at in the current state of art, and what problem is still challenging. Essentially, the classes of problems are (no my list is definitely not exhaustive):

a.   image recognition and classification
b.   natural language processing:   translation and 
c.   audio:   speech recognition

And what problem it is still a challenge:

a.   logic processing / understanding and proving
b.   source code processing:  automated programming, bug fixing, bugs finding etc.
c.   problem diagnosis (eg, engine, or mechanical problem diagnosis).

My knowledge is limited, eg, the last item above may well be very much advanced right now, as it has a long history starting from the use of experts system for problem diagnosis (and had many success stories).

So like (c) above, you have to frame your problems in the form that fits one of the above well-known domain or problem classes, before even identifying the algorithm.

Broadly, algorithm can be broadly classified as follows:


enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.