Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I'm using Keras on top of Theano for neural network training. What should be my batch size in relation to the number of classes? I have 560 classes and if I use a batch size more than 128, I can't train as it does not fit in memory. Would it help to have the batch size greater than the number of classes, say,
batch size = 3 * number of classes?
That way each batch has at least a few images from each class. I do understand there is a randomness in selecting data points for each batch.
The batch size you choose can have an impact on the training of your neural network, and it's generally recommended to have a batch size that is larger than the number of classes. This is because a larger batch size allows for more data to be processed at once, which can lead to more accurate gradients and faster convergence. However, as you've mentioned, there is a trade-off between batch size and memory usage.
In your case, it seems that using a batch size of 3 * number of classes (3 * 560 = 1680) may not be feasible due to memory constraints. One strategy you could try is to use a smaller batch size, such as 128, and compensate by increasing the number of iterations or epochs. This will increase the amount of data seen by the model over time, and can still lead to good performance.
Another approach to consider is using a technique called "stochastic gradient descent with restarts" (SGDR) which allows to increase the learning rate over time and can help to converge faster.
It's also important to keep in mind that there is randomness in selecting data points for each batch, so it's still possible for a batch to contain a good representative sample of each class even with a smaller batch size. And also, it's worth to mention that, the overall number of classes is not the only factor that determines the best batch size, factors such as the number of samples per class, the complexity of the model, and the size of the input images can also play a role in determining the optimal batch size.
