What kind of deep neural networks are (not) data-intensive?

Question

There are plenty of shapes and tastes of neural networks out there. Just like any machine learning model they require as much data you can get to deliver good performance, but it seems that some models are more data-intensive than others. I looked for an overview of which models require more/less data to work well, but failed to find anything. I was also wondering what kind of tricks can be used to make neural networks less data-intensive.

The things I learnt thus far from different sources:

1. Deep Belief Networks require less labelled data, because pre-training can be done with unlabelled data, which is often easier to get
2. Sigmoid activations suffer from the vanishing gradient problem and therefore require more data than newer activation functions like tanh, ReLU and ELU
3. Recurrent Neural Networks require more data than other models

Can these statements be confirmed? Does anybody have a good reference to an overview or the knowledge to provide one?

Answer 1

If you take a careful look at the recent state-of-the-art results on some standard problems (language models, computer vision), you'll notice that the number of parameters in a network scales with the number of examples in the training data. (This is not the main thesis of this paper, but it is discussed in a few paragraphs of "mixup: Beyond Empirical Risk Minimization" by Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, David Lopez-Paz.) Stated another way, you can model a small data set with a small model.

Or, looking at the problem from the other end, for any network which achieves state-of-the-art results for some problem, you could just remove any number of parameters and accept the drop in performance as the cost if training a smaller network.