I have been doing work on Theano based autoencoder. For data size of less than 100, it is working perfectly using batch gradient descent. But for data size around 500, it is better to use mini batch gradient descent I think. Because it takes long time for training. So, What is the minimum number of samples required for mini batch gradient descent ? Any suggestions about mini batch size also ? The error is less for batch gradient descent than mini batch gradients (mini_batch_size :25, n_epochs : 200).
$\begingroup$
$\endgroup$
5
-
$\begingroup$ The minimum mini batch size is 1 (actually some algorithms even sometimes using a min batch size of 0, i.e., reusing old gradient data rather than drawing a gradient sample at the current point). As to what will work best, well that's that's a different matter, and there seems to be a lot of art and trial and error involved. The optimal mini batch size can be crucially dependent on small little details of the overall algorithm being used, as well as problem class, etc. If there were some well-accepted universal rules for choosing optimal min-batch sizes, you likely would have already seen it. $\endgroup$– Mark L. StoneCommented Dec 5, 2016 at 16:06
-
$\begingroup$ Yes. I agree there is no universal rules like that. Thank you. But in my case, actually when I train 500 samples using batch descent, the error is very less than training it using mini batch size of 25, for 200 epochs. I just want to confirm whether my model with mini batch is working properly or not. $\endgroup$– ShyamkkhadkaCommented Dec 5, 2016 at 17:48
-
$\begingroup$ Personally, for problems I work on, I hate stochastic gradient descent, and mini-batch sizes of 1 or a small number. I prefer to work with larger mini-batch (number of simulation replications) sizes (but it depends on variance of objective function and gradient) and my own algorithms (see, for example twitter.com/UltraSimOpt ) automatically adapt mini-batch (sample) size based on what "they see", generally increasing as the optimization progresses. However, the goodness of doing so is heavily dependent on details of the entirety of the optimization algorithm. $\endgroup$– Mark L. StoneCommented Dec 5, 2016 at 19:10
-
$\begingroup$ Especially if the objective function is non-convex, you are likely to see more benefit from larger mini-batch sizes, and small mini-batch sizes might not work well. Even for convex objective function, larger mini-batch sizes might work out much better than small mini-batch sizes. So your experience does not mean your algorithm implementation is incorrect (by the way, I'm not saying that it is correct, either). Variability (uncertainty) of objective function and gradient relative to (projected) gradient is key, and plays big role in optimal mini-batch sizes. $\endgroup$– Mark L. StoneCommented Dec 5, 2016 at 19:18
-
$\begingroup$ @MarkL.Stone Would you consider collecting these thoughts in an answer on the linked thread? I think this is a valuable contribution and I'd hate for it to get lost in a comment thread. $\endgroup$– Sycorax ♦Commented Dec 6, 2016 at 20:08
Add a comment
|