I am new to deep learning, so this might be a trivial question. But I am wondering why deep learning (or neural network) does not work very well on small labeled data. Whatever research papers I have read, their datasets are huge. Intuitively that's not surprising because our brain takes a lot of time to train itself. But is there a mathematical proof or reason why neural network does not work well in such cases ?


The neural networks used in typical deep learning models have a very large number of nodes with many layers, and therefore many parameters that must be estimated. This requires a lot of data. A small neural network (with fewer layers and fewer free parameters) can be successfully trained with a small data set - but this would not usually be described as "deep learning".

  • 3
    $\begingroup$ +1. Model complexity should always only grow slowly with sample size, and deep learning is a pretty complex model, implying that it will usually not work well for small sample sizes. The Elements of Statistical Learning (available for download for free) discusses this - highly recommended. $\endgroup$ Jul 22 '15 at 11:59
  • $\begingroup$ Thanks. Does that mean that if I still try to learn a model using small data, I am going to overfit the model ? $\endgroup$
    – bluechill
    Jul 22 '15 at 17:03
  • 3
    $\begingroup$ You are more likely to overfit if you have a small amount of data relative to the number of parameters in your model -- this is true for any model. You can add regularizers (e.g., penalize large weights, add noise to input data, drop out hidden units, etc.) to your model to help avoid this, but it's sort of an art rather than a science at the moment. $\endgroup$
    – lmjohns3
    Jul 23 '15 at 15:18

This site is temporarily in read only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .