14
$\begingroup$

If I have a convolutional neural network (CNN), which has about 1,000,000 parameters, how many training data is needed (assume I am doing stochastic gradient descent)? Is there any rule of thumb?

Additional notes: When I performed stochastic gradient descent (e.g., 64 patches for 1 iteration), after ~10000 iterations, the accuracy of the classifier can reach to a rough steady value). Is this mean not many data is needed? Like 100k-1000k data.

$\endgroup$
11
$\begingroup$

In order to figure out whether or not more data will be helpful, you should compare the performance of your algorithm on the training data (i.e. the data used to train the neural network) to its performance on testing data (i.e. data the neural network did not "see" in training).

A good thing to check would be the error (or accuracy) on each set as a function of iteration number. There are two possibilities for the outcome of this:

1) The training error converges to a value significantly lower than than the testing error. If this is the case, the performance of your algorithm will almost certainly improve with more data.

2) The training error and the testing error converge to about the same value (with the training error still probably being slightly lower than the testing error). In this case additional data by itself will not help your algorithm. If you need better performance than you are getting at this point, you should try either adding more neurons to your hidden layers, or adding more hidden layers. If enough hidden units are added, you will find your testing error will become noticeably higher than the training error, and more data will help at that point.

For a more thorough and helpful introduction to how to make these decisions, I highly recommend Andrew Ng's Coursera course, particularly the "Evaluating a learning algorithm" and "Bias vs. Variance" lessons.

$\endgroup$
3
$\begingroup$

The naive answer is that always more data are needed. Iterating over the same dataset saying for more epochs helps you to "refine" the result but you don't improve the result as much as having more data.

As an example i'm training a convnet to do sentence modelling and to test if i need more data i tried to split my training dataset in smaller subset and trying to test it.

Using the whole dataset and training for 10 iteration i obtained 93% accuracy on my benchmark and it keep improving. Instead when i iterated on the 10% of the dataset for 100 iteration i obtained a 85%.

So always try to have more data but if you can't, doing more epochs can be a nice trade-of but in the end your model converges better if you fed the network with always new data.

$\endgroup$
  • 1
    $\begingroup$ Thanks for your answer! Yes, I know the more the data you have, the better you will be, but then exactly how many data you need is what I am not sure. Because not always you can have more data. For example, I need to hire someone to do annotation and labeling, I want to figure out when I should stop... $\endgroup$ – RockTheStar Dec 9 '14 at 18:53
  • $\begingroup$ does that mean if i have a small dataset my model will not converge? $\endgroup$ – samsamara Sep 16 '16 at 6:54
3
$\begingroup$

I guess the most important thing is that the samples in your data are well spread, because no matter how much data you have, more data would always be better. After all, if you try to learn to distinguish between cat and dog pictures, you can't expect your model to perform well if you only feed it cat images.

As suggested in the answer by Kevin L, it makes sense to consider the difference between training error and testing error. If your testing data is independent from your training data, this gives an indication as to how well your model generalises to the data that is unavailable. Something I would like to add to that is the fact that a large difference between training and testing error only tells you that your model does not generalise well, i.e. you are overfitting on the training data. More data will probably help, because now the network also needs to model the extra data points, hence cannot overfit that much anymore. However, it might be more worthwhile to change your model such that it generalises better. This chapter from an excellent book explains what types of regularisation exist and how they can be applied in networks to get better generalisation.

If you were looking for a more quantitive measure, I recently found this question on quora. It is about an auto-encoder, but I guess it should also be applicable to your example. I have no idea whether this is correct (please let me know), but I would reason that for instance for MNIST, one could argue that you try to reduce images with a maximum of 28 * 28 * 8 * 10 000 = 62 720 000 bits entropy to ten classes in one-hot encoding with 10 * 10 * 10 000 = 1 000 000 bits of entropy. Because we are only interested in the 1 000 000 bits of entropy at the output, we can say that with 1 000 000 parameters, each parameter represents a single bit, which is 1e-4 bit per sample. This means you would need more data. Or you have too much parameters, because e.g. with 100 parameters, you have 10 000 bits per parameter and therefore 1 bit per sample. However, I would like to emphasise once again that it is the first time I have seen something like this and if anyone could confirm this, I would be grateful.

$\endgroup$
  • $\begingroup$ Thanks for your suggestion. I think nowadays people come-up with different strategy to reduce the number of data for a specific task. $\endgroup$ – RockTheStar Dec 6 '16 at 16:35
0
$\begingroup$

Another method generally used to figure out if your network has learned enough features is to visualize the initial filters. If the network is well trained it should display a smooth filter. A noisy filter generally indicates that the network hasn't been trained enough or that it has been overfit. For more info read this page.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.