Any rule of thumb, on how many input images would be needed to have a reasonable chance not to overfit the data when training a CNN from scratch?

In other words, what is a reasonable amount of data (in order of magnitude) needed to train a CNN from scratch?

For instance, for ResNet 50 that has about 26M parameters.

  • 1
    $\begingroup$ Some theoretical work from NIPS 2018 in paper "How Many Samples are Needed to Estimate a Convolutional Neural Network"? papers.nips.cc/paper/… $\endgroup$ – itdxer Jan 15 at 9:40

There's no real rule of thumb to this, as it highly depends on the classification/regression problem and the nature of your images. Generally speaking, you need thousands, but usually, orders of magnitude more. There are smaller examples, e.g. the LUNA16 lung nodule detection challenge only has around 1000 images.. When your data is small, you hopefully have the ability to either heavily regularize the model, or provide plenty of initial image transformations (shifts, scalings, etc.).

A nice reference is "Understanding Deep Learning Requires Rethinking Generalization", where the authors show that a typical CNN can easily memorize all of ImageNet (A million images) even when the labels are totally random, i.e. training error goes to 0. So 30 million parameters is plenty for memorizing huge amounts of data.

With that being said, you're much more likely to get better results by not starting from scratch, instead opting to use a pretrained CNN (on say, ImageNet), as the bulk of the filters it learns tend to generalize quite well to other tasks, and will greatly reduce overfitting in the earlier stages of your training.

To be clear, there is almost no reason to start from complete scratch. There's a great example of this from the 20bn-kitchenware dataset, where the task is to predict whether actions are successful (e.g. did the person pick up the cup successfully, or fail?). This paper: "The more fine-grained, the better for transfer learning" - Mahdisoltani, et. al. has a nice summary in Figure 4 of baselines tried for this, including pertaining on ImageNet and pretaining on predicting whether videos are running forwards or backwards. The forwards-backwards pertaining was a brilliant inspiration, because it not only teaches the CNN to recognize cause-and-effect, but also inadvertently teaches the CNN about how gravity works!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.