4
$\begingroup$

After searching I read diferent theorys that using a greater batch size has better performance while model is training, but in the other hand, I also find the oposite view, that using a mini-batch size, like the default one (32), have good results in general. I also think that this have a relation between the amount of input data your are going to use for train the model and the number of epochs. I know this is quite dificult to know because of the uncertainty that set out the problem you are working on.

But is there any "rule" or "logic" that establish a minimun guide or something like that?

$\endgroup$
1
  • 1
    $\begingroup$ If you're training a triplet network using hard negative mining, then there are very good reasons to believe that a larger batch size is better. stats.stackexchange.com/questions/475655/… $\endgroup$
    – Sycorax
    Commented May 31, 2021 at 13:57

1 Answer 1

6
$\begingroup$

How to choose a batch size

The short answer is that batch size itself can be considered a hyperparameter, so experiment with training using different batch sizes and evaluate the performance for each batch size on the validation set.

The long answer is that the effect of different batch sizes is different for every model. For example, when using GPU acceleration, training can physically become faster if you increase your batch size until you saturate GPU load. Decreasing batch size can also affect performance if your network has BatchNorm layers, it might improve your network's performance or it might decrease it.

There is no hard and fast rule when it comes to a network of your own design, and more often than not I find it common (both anecdotally and in literature) that for large convolutional nets the batch size is chosen to be the biggest that can actually fit on the GPU.


How to chose number of epochs while training a NN

The answer here is early stopping. Instead of 'choosing' a number of epochs you instead save the network weights from the 'best' epoch. This optimal epoch is determined by validation loss. After each epoch you predict on the validation set and calculate the loss. Whenever the validation loss after an epoch beats the previous best (i.e. is lower) you checkpoint network state, overwriting the previous checkpoint made at the previous 'best' epoch. If the validation loss doesn't improve after, for example, 10 epochs you can stop training.

Tensorflow has built-in support for early stopping in Keras for both the sequential API and functional API, and if you're using a custom training loop you can implement the early stopping logic quite easily yourself.

$\endgroup$
1
  • $\begingroup$ Thanks! How to determine best epoch in case of unsupervised learning? $\endgroup$ Commented Oct 18, 2021 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.