There are plenty of shapes and tastes of neural networks out there. Just like any machine learning model they require as much data you can get to deliver good performance, but it seems that some models are more data-intensive than others. I looked for an overview of which models require more/less data to work well, but failed to find anything. I was also wondering what kind of tricks can be used to make neural networks less data-intensive.
The things I learnt thus far from different sources:
- Deep Belief Networks require less labelled data, because pre-training can be done with unlabelled data, which is often easier to get
- Sigmoid activations suffer from the vanishing gradient problem and therefore require more data than newer activation functions like tanh, ReLU and ELU
- Recurrent Neural Networks require more data than other models
Can these statements be confirmed? Does anybody have a good reference to an overview or the knowledge to provide one?