Linked Questions
21 questions linked to/from What is the trade-off between batch size and number of iterations to train a neural network?
2
votes
0
answers
196
views
How to choose a good batch size for a powerful CPU [duplicate]
I am training a deep neural networks for self driving cars using Adam optimization, and I wonder how can I find a standard batch size value , currently I am using the value 1 and I can see that my ...
- 21
2
votes
0
answers
95
views
Choosing an ideal minibatch size for a large sample [duplicate]
I know that bigger batch size gives more accurate results, but I'm not sure which batch size is ideal given the following cases:
Training on 65000 examples and validating on 13000 examples
Training ...
- 21
0
votes
0
answers
87
views
minimum data size required for good performance of mini batch gradient descent [duplicate]
I have been doing work on Theano based autoencoder. For data size of less than 100, it is working perfectly using batch gradient descent. But for data size around 500, it is better to use mini batch ...
- 253
299
votes
6
answers
688k
views
What is batch size in neural network?
I'm using Python Keras package for neural network. This is the link. Is batch_size equals to number of test samples? From ...
- 3,951
150
votes
6
answers
170k
views
Batch gradient descent versus stochastic gradient descent
Suppose we have some training set $(x_{(i)}, y_{(i)})$ for $i = 1, \dots, m$. Also suppose we run some type of supervised learning algorithm on the training set. Hypotheses are represented as $h_{\...
- 1,541
55
votes
1
answer
92k
views
How large should the batch size be for stochastic gradient descent?
I understand that stochastic gradient descent may be used to optimize a neural network using backpropagation by updating each iteration with a different sample of the training dataset. How large ...
- 2,101
36
votes
4
answers
47k
views
How does batch size affect convergence of SGD and why?
I've seen similar conclusion from many discussions, that as the minibatch size gets larger the convergence of SGD actually gets harder/worse, for example this paper and this answer. Also I've heard of ...
- 15.9k
9
votes
2
answers
39k
views
How does batch size affect Adam Optimizer?
What impact does mini-batch size have on Adam Optimizer? Is there a recommended mini-batch size when training a (covolutional) neural network with Adam Optimizer?
From what I understood (I might be ...
- 319
12
votes
3
answers
7k
views
Neural Networks: Is an epoch in SGD the same as an epoch in mini-batch?
In SGD an epoch would be the full presentation of the training data, and then there would be N weight updates per epoch (if there are N data examples in the training set).
If we now do mini-batches ...
- 121
14
votes
1
answer
28k
views
Deep Learning: Why does increase batch_size cause overfitting and how does one reduce it?
I used to train my model on my local machine, where the memory is only sufficient for 10 examples per batch. However, when I migrated my model to AWS and used a bigger GPU (Tesla K80), I could ...
- 1,683
13
votes
2
answers
12k
views
Minimum Training size for simple neural net
There's an old rule of thumb for multivariate statistics that recommends a minimum of 10 cases for each independent variable. But that's often where there is one parameter to fit for each variable.
...
- 151
9
votes
2
answers
10k
views
Too large batch size
I experiment with CIFA10 datasets. With my model I found that the larger the batch size, the better the model can learn the dataset. From what I see on the internet the typical size is 32 to 128, and ...
- 1,565
8
votes
1
answer
5k
views
Can small SGD batch size lead to faster overfitting?
I have feedforward neural net, trained on cca 34k samples and tested on 8k samples. There is 139 features in dataset. The ANN does classification between two labels, 0 and 1, so I am using sigmoid ...
- 301
6
votes
1
answer
5k
views
Online learning in LSTM
Recently, I have been working on RNNs (LSTM specifically) to do time series prediction and I have used different frameworks such as deeplearning4j and theano (keras). As you may know, one of the ...
- 366
5
votes
1
answer
2k
views
Training batch size in relation to number of classes in a neural network
I'm using Keras on top of Theano for neural network training.
What should be my batch size in relation to the number of classes? I have 560 classes and if I use a batch size more than 128, I can't ...
- 151