Linked Questions
21 questions linked to/from What is the trade-off between batch size and number of iterations to train a neural network?
2
votes
0
answers
210
views
How to choose a good batch size for a powerful CPU [duplicate]
I am training a deep neural networks for self driving cars using Adam optimization, and I wonder how can I find a standard batch size value , currently I am using the value 1 and I can see that my ...
- 21
2
votes
0
answers
106
views
Choosing an ideal minibatch size for a large sample [duplicate]
I know that bigger batch size gives more accurate results, but I'm not sure which batch size is ideal given the following cases:
Training on 65000 examples and validating on 13000 examples
Training ...
- 21
0
votes
0
answers
89
views
minimum data size required for good performance of mini batch gradient descent [duplicate]
I have been doing work on Theano based autoencoder. For data size of less than 100, it is working perfectly using batch gradient descent. But for data size around 500, it is better to use mini batch ...
- 287
305
votes
6
answers
738k
views
What is batch size in neural network?
I'm using Python Keras package for neural network. This is the link. Is batch_size equals to number of test samples? From ...
- 4,271
154
votes
7
answers
175k
views
Batch gradient descent versus stochastic gradient descent
Suppose we have some training set $(x_{(i)}, y_{(i)})$ for $i = 1, \dots, m$. Also suppose we run some type of supervised learning algorithm on the training set. Hypotheses are represented as $h_{\...
- 1,581
55
votes
1
answer
95k
views
How large should the batch size be for stochastic gradient descent?
I understand that stochastic gradient descent may be used to optimize a neural network using backpropagation by updating each iteration with a different sample of the training dataset. How large ...
- 2,121
36
votes
4
answers
50k
views
How does batch size affect convergence of SGD and why?
I've seen similar conclusion from many discussions, that as the minibatch size gets larger the convergence of SGD actually gets harder/worse, for example this paper and this answer. Also I've heard of ...
- 16.8k
12
votes
2
answers
44k
views
How does batch size affect Adam Optimizer?
What impact does mini-batch size have on Adam Optimizer? Is there a recommended mini-batch size when training a (covolutional) neural network with Adam Optimizer?
From what I understood (I might be ...
- 359
18
votes
1
answer
33k
views
Deep Learning: Why does increase batch_size cause overfitting and how does one reduce it?
I used to train my model on my local machine, where the memory is only sufficient for 10 examples per batch. However, when I migrated my model to AWS and used a bigger GPU (Tesla K80), I could ...
- 1,783
12
votes
3
answers
8k
views
Neural Networks: Is an epoch in SGD the same as an epoch in mini-batch?
In SGD an epoch would be the full presentation of the training data, and then there would be N weight updates per epoch (if there are N data examples in the training set).
If we now do mini-batches ...
- 121
13
votes
2
answers
14k
views
Minimum Training size for simple neural net
There's an old rule of thumb for multivariate statistics that recommends a minimum of 10 cases for each independent variable. But that's often where there is one parameter to fit for each variable.
...
- 151
10
votes
2
answers
11k
views
Too large batch size
I experiment with CIFA10 datasets. With my model I found that the larger the batch size, the better the model can learn the dataset. From what I see on the internet the typical size is 32 to 128, and ...
- 1,655
8
votes
1
answer
5k
views
Can small SGD batch size lead to faster overfitting?
I have feedforward neural net, trained on cca 34k samples and tested on 8k samples. There is 139 features in dataset. The ANN does classification between two labels, 0 and 1, so I am using sigmoid ...
- 301
6
votes
1
answer
5k
views
Online learning in LSTM
Recently, I have been working on RNNs (LSTM specifically) to do time series prediction and I have used different frameworks such as deeplearning4j and theano (keras). As you may know, one of the ...
- 366
5
votes
1
answer
2k
views
Training batch size in relation to number of classes in a neural network
I'm using Keras on top of Theano for neural network training.
What should be my batch size in relation to the number of classes? I have 560 classes and if I use a batch size more than 128, I can't ...
- 151
4
votes
1
answer
608
views
Reducing batch size after X epochs?
Batch size and the number of iterations are considered as a tradeoff.
It has been observed in practice that when using a larger batch there is a significant degradation in the quality of the model, ...
- 151
6
votes
2
answers
152
views
Conventional wisdom for designing and training neural networks
Are there any tutorials or guides on conventional wisdom for designing neural networks? For example, how do you pick:
the number of layers or number of units per layer?
an activation function?
a step ...
- 1,277
1
vote
1
answer
191
views
neural network for data set with large number of samples
What are the rules of thumb for neural network configurations with large number of samples? I have a dataset with 200k samples, 400 features, and binary label classification problem. How should I ...
- 111
3
votes
0
answers
224
views
Is mini-batch / stochastic gradient descend similar implicitly adding the same effect as simulated annealing?
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak Peter Tang. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. https://arxiv.org/...
- 564
3
votes
1
answer
78
views
Can you perform stochastic learning followed by batch learning in neural networks?
I'm trying to teach myself about neural networks. I've been reading through the "Efficient BackProp" paper that's highly sited and it's brought me this question;
Since stochastic learning converges ...
- 233
1
vote
1
answer
40
views
Is there any other reason for using SGD than reducing time until convergence?
Is there any other reason for using Stochastic Gradient Descent than reducing time until convergence? In other words, does it ever make sense to try out SGD when regular Gradient Descent runs fairly ...
- 171