Is there any supervised-learning problem that (deep) neural networks obviously couldn't outperform any other methods? I have seen people have put a lot of efforts on SVM and Kernels, and they look pretty interesting as a starter in Machine Learning. But if we expect that almost-always we could find outperforming solution in terms of (deep) Neural Network, what is the meaning of trying other methods in this era?
Here is my constraint on this topic.


*

*We think of only Supervised-Learnings; Regression, and Classification.

*Readability of the Result is not counted; only the Accuracy on the Supervised-Learning Problem counts.

*Computational-Cost is not in consideration.

*I am not saying that any other methods are useless.

 A: Here  is one theoretical and two practical reasons why someone might rationally prefer a non-DNN approach. 


*

*The No Free Lunch Theorem from Wolpert and Macready says

We have dubbed the associated results NFL theorems because they demonstrate that if an algorithm performs well on a certain class of problems then it necessarily pays for that with degraded performance on the set of all remaining problems.

In other words, no single algorithm rules them all; you've got to benchmark. 
The obvious rebuttal here is that you usually don't care about all possible problems, and deep learning seems to work well on several classes of problems that people do care about (e.g., object recognition), and so it's a reasonable first/only choice for other applications in those domains.

*Many of these very deep networks require tons of data, as well as tons of computation, to fit. If you have (say) 500 examples, a twenty layer network is never going to learn well, while it might be possible to fit a much simpler model. There are a surprising number of problems where it's not feasible to collect a ton of data. On the other hand, one might try learning to solve a related problem (where more data is available), use something like transfer learning to adapt it to the specific low-data-availability-task.

*Deep neural networks can also have unusual failure modes. There are some papers showing that barely-human-perceptible changes can cause a network to flip from correctly classifying an image to confidently misclassifying it. (See here and the accompanying paper by Szegedy et al.) Other approaches may be more robust against this: there are poisoning attacks against SVMs (e.g., this by Biggio, Nelson, and Laskov), but those happen at train, rather than test time. At the opposite extreme, there are known (but not great) performance bounds for the nearest-neighbor algorithm. In some situations, you might happier with lower overall performance with less chance of catastrophe. 
A: Somewhere on this playlist of lectures by Geoff Hinton (from his Coursera course on neural networks), there's a segment where he talks about two classes of problems:


*

*Problems where noise is the key feature,

*Problems where signal is the key feature.


I remember the explanation that while neural nets thrive in this latter space, traditional statistical methods are often better suited to the former. Analyzing high-res digital photographs of actual things in the world, a place where deep convolutional nets excel, clearly constitutes the latter. 
On the other hand, when noise is the dominant feature, for example, in a medical case-control study with 50 cases and 50 controls, traditional statistical methods may be better suited to the problem.
If anybody finds that video, please comment and I'll update. 
A: Two linearly perfected correlated variables. Can deep-network with 1 million hidden layers and 2 trillion neutrons beat a simple linear regression?
EDITED
In my experience, sample collection is more expensive than computation. I mean, we can just hire some Amazon instances, run deep learning training and then come back a few days later. The cost in my field is about $200 USD. The cost is minimal. My colleagues earn more than that in a day.
Sample collection generally requires domain knowledge and specialized equipments. Deep learning is only suitable for problems with cheap and easy access data set, such as natural language processing, image processing and anything that you can scrape off from the Internet.
A: To be honest, it is not possible for a deep-learning method to outperform a kernel methods. Why ? It is very simple, because any network, be it deep or shallow, can be described by a kernel. Thus a kernel can reproduce any result coming from deep learning. However, a kernel method have access to other, more powerful methods, than deep learning ones. Indeed, today, kernel machines obtains results that are far better than any deep learning approach.
EDIT : As I received a warning concerning this answer, please let me detail it.
I reference this paper : https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3769804.

*

*Every neural network (NN) can be input in a kernel machine, it is known since more than a decade now. For instance, in our kernel machines, we can input any NNs, there is an interface to it, see section 2.3.7.

*A kernel machine have access to other methods than NNs. See for instance section 3, that allows any PDE approach.

*We benchmarked both approaches. I am sorry, but our kernel machines always retrieved better results than NN ones, as for instance the MNIST test in  https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3766451
