If I have a convolutional neural network (CNN), which has about 1,000,000 parameters, how many training data is needed (assume I am doing stochastic gradient descent)? Is there any rule of thumb?
Additional notes: When I performed stochastic gradient descent (e.g., 64 patches for 1 iteration), after ~10000 iterations, the accuracy of the classifier can reach to a rough steady value). Is this mean not many data is needed? Like 100k-1000k data.
In order to figure out whether or not more data will be helpful, you should compare the performance of your algorithm on the training data (i.e. the data used to train the neural network) to its performance on testing data (i.e. data the neural network did not "see" in training).
A good thing to check would be the error (or accuracy) on each set as a function of iteration number. There are two possibilities for the outcome of this:
1) The training error converges to a value significantly lower than than the testing error. If this is the case, the performance of your algorithm will almost certainly improve with more data.
2) The training error and the testing error converge to about the same value (with the training error still probably being slightly lower than the testing error). In this case additional data by itself will not help your algorithm. If you need better performance than you are getting at this point, you should try either adding more neurons to your hidden layers, or adding more hidden layers. If enough hidden units are added, you will find your testing error will become noticeably higher than the training error, and more data will help at that point.
For a more thorough and helpful introduction to how to make these decisions, I highly recommend Andrew Ng's Coursera course, particularly the "Evaluating a learning algorithm" and "Bias vs. Variance" lessons.
I guess the most important thing is that the samples in your data are well spread, because no matter how much data you have, more data would always be better. After all, if you try to learn to distinguish between cat and dog pictures, you can't expect your model to perform well if you only feed it cat images.
As suggested in the answer by Kevin L, it makes sense to consider the difference between training error and testing error. If your testing data is independent from your training data, this gives an indication as to how well your model generalises to the data that is unavailable. Something I would like to add to that is the fact that a large difference between training and testing error only tells you that your model does not generalise well, i.e. you are overfitting on the training data. More data will probably help, because now the network also needs to model the extra data points, hence cannot overfit that much anymore. However, it might be more worthwhile to change your model such that it generalises better. This chapter from an excellent book explains what types of regularisation exist and how they can be applied in networks to get better generalisation.
If you were looking for a more quantitive measure, I recently found this question on quora. It is about an auto-encoder, but I guess it should also be applicable to your example. I have no idea whether this is correct (please let me know), but I would reason that for instance for MNIST, one could argue that you try to reduce images with a maximum of 28 * 28 * 8 * 10 000 = 62 720 000 bits entropy to ten classes in one-hot encoding with 10 * 10 * 10 000 = 1 000 000 bits of entropy. Because we are only interested in the 1 000 000 bits of entropy at the output, we can say that with 1 000 000 parameters, each parameter represents a single bit, which is 1e-4 bit per sample. This means you would need more data. Or you have too much parameters, because e.g. with 100 parameters, you have 10 000 bits per parameter and therefore 1 bit per sample. However, I would like to emphasise once again that it is the first time I have seen something like this and if anyone could confirm this, I would be grateful.
The naive answer is that always more data are needed. Iterating over the same dataset saying for more epochs helps you to "refine" the result but you don't improve the result as much as having more data.
As an example i'm training a convnet to do sentence modelling and to test if i need more data i tried to split my training dataset in smaller subset and trying to test it.
Using the whole dataset and training for 10 iteration i obtained 93% accuracy on my benchmark and it keep improving. Instead when i iterated on the 10% of the dataset for 100 iteration i obtained a 85%.
So always try to have more data but if you can't, doing more epochs can be a nice trade-of but in the end your model converges better if you fed the network with always new data.
Another method generally used to figure out if your network has learned enough features is to visualize the initial filters. If the network is well trained it should display a smooth filter. A noisy filter generally indicates that the network hasn't been trained enough or that it has been overfit. For more info read this page.
