There are plenty of shapes and tastes of neural networks out there. Just like any machine learning model they require as much data you can get to deliver good performance, but it seems that some models are more data-intensive than others. I looked for an overview of which models require more/less data to work well, but failed to find anything. I was also wondering what kind of tricks can be used to make neural networks less data-intensive.

The things I learnt thus far from different sources:

  1. Deep Belief Networks require less labelled data, because pre-training can be done with unlabelled data, which is often easier to get
  2. Sigmoid activations suffer from the vanishing gradient problem and therefore require more data than newer activation functions like tanh, ReLU and ELU
  3. Recurrent Neural Networks require more data than other models

Can these statements be confirmed? Does anybody have a good reference to an overview or the knowledge to provide one?

  • $\begingroup$ This is a very general question and will probably not get answer anytime soon for a number of reasons. While the three things you mentioned in your question are generally correct, model architecture is the MOST important factor in its data-intensivity. $\endgroup$ – NULL Aug 1 '17 at 12:27
  • $\begingroup$ The deeper the model, the more complex the optimization surface. That'll affect amount of data needed. Also, the higher the number of weight/paramters to tune, the higher the number of samples. This is similar to the previous point though. $\endgroup$ – NULL Aug 1 '17 at 12:29
  • $\begingroup$ You'd probably have a better chance of getting an informative answer if you breakdown your question to smaller more specific (model specific) ones. $\endgroup$ – NULL Aug 1 '17 at 12:29

If you take a careful look at the recent state-of-the-art results on some standard problems (language models, computer vision), you'll notice that the number of parameters in a network scales with the number of examples in the training data. (This is not the main thesis of this paper, but it is discussed in a few paragraphs of "mixup: Beyond Empirical Risk Minimization" by Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, David Lopez-Paz.) Stated another way, you can model a small data set with a small model.

Or, looking at the problem from the other end, for any network which achieves state-of-the-art results for some problem, you could just remove any number of parameters and accept the drop in performance as the cost if training a smaller network.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.