I haven't found a satisfactory answer to this from google.
Of course if the data I have is of the order of millions then deep learning is the way.
And I have read that when I do not have big data then maybe it is better to use other methods in machine learning. The reason given is over-fitting. Machine learning: i.e. looking at data, feature extractions, crafting new features from what is collected etc. things such as removing heavily correlated variables etc. the whole machine learning 9 yards.
And I have been wondering: why is it that the neural networks with one hidden layer are not panacea to machine learning problems? They are universal estimators, over-fitting can be managed with dropout, l2 regularization, l1 regularization, batch-normalization. Training speed is not generally an issue if we have just 50,000 training examples. They are better at test time than, let us say, random forests.
So why not - clean the data, impute missing values as you would generally do, center the data, standardize the data, throw it to an ensemble of neural networks with one hidden layer and apply regularization till you see no over-fitting and then train them to the end. No issues with gradient explosion or gradient vanishing since it is just a 2 layered network. If deep layers were needed, that means hierarchical features are to be learned and then other machine learning algorithms are no good as well. For example SVM is a neural network with hinge loss only.
An example where some other machine learning algorithm would outperform a carefully regularized 2 layered (maybe 3?) neural network would be appreciated. You can give me the link to the problem and I would train the best neural network that I can and we can see if 2 layered or 3 layered neural networks falls short of any other benchmark machine learning algorithm.