Linked Questions

2
votes
0answers
71 views

How to choose a good batch size for a powerful CPU [duplicate]

I am training a deep neural networks for self driving cars using Adam optimization, and I wonder how can I find a standard batch size value , currently I am using the value 1 and I can see that my ...
0
votes
0answers
68 views

minimum data size required for good performance of mini batch gradient descent [duplicate]

I have been doing work on Theano based autoencoder. For data size of less than 100, it is working perfectly using batch gradient descent. But for data size around 500, it is better to use mini batch ...
2
votes
0answers
39 views

Choosing an ideal minibatch size for a large sample [duplicate]

I know that bigger batch size gives more accurate results, but I'm not sure which batch size is ideal given the following cases: Training on 65000 examples and validating on 13000 examples Training ...
171
votes
4answers
244k views

What is batch size in neural network?

I'm using Python Keras package for neural network. This is the link. Is batch_size equals to number of test samples? From ...
100
votes
3answers
90k views

Batch gradient descent versus stochastic gradient descent

Suppose we have some training set $(x_{(i)}, y_{(i)})$ for $i = 1, \dots, m$. Also suppose we run some type of supervised learning algorithm on the training set. Hypotheses are represented as $h_{\...
49
votes
1answer
67k views

How large should the batch size be for stochastic gradient descent?

I understand that stochastic gradient descent may be used to optimize a neural network using backpropagation by updating each iteration with a different sample of the training dataset. How large ...
17
votes
3answers
23k views

How does batch size affect convergence of SGD and why?

I've seen similar conclusion from many discussions, that as the minibatch size gets larger the convergence of SGD actually gets harder/worse, for example this paper and this answer. Also I've heard of ...
8
votes
2answers
2k views

Neural Networks: Is an epoch in SGD the same as an epoch in mini-batch?

In SGD an epoch would be the full presentation of the training data, and then there would be N weight updates per epoch (if there are N data examples in the training set). If we now do mini-batches ...
2
votes
1answer
6k views

How does batch size affect Adam Optimizer?

What impact does mini-batch size have on Adam Optimizer? Is there a recommended mini-batch size when training a (covolutional) neural network with Adam Optimizer? From what I understood (I might be ...
7
votes
1answer
6k views

Deep Learning: Why does increase batch_size cause overfitting and how does one reduce it?

I used to train my model on my local machine, where the memory is only sufficient for 10 examples per batch. However, when I migrated my model to AWS and used a bigger GPU (Tesla K80), I could ...
7
votes
2answers
1k views

Minimum Training size for simple neural net

There's an old rule of thumb for multivariate statistics that recommends a minimum of 10 cases for each independent variable. But that's often where there is one parameter to fit for each variable. ...
3
votes
1answer
3k views

Too large batch size

I experiment with CIFA10 datasets. With my model I found that the larger the batch size, the better the model can learn the dataset. From what I see on the internet the typical size is 32 to 128, and ...
4
votes
1answer
2k views

Online learning in LSTM

Recently, I have been working on RNNs (LSTM specifically) to do time series prediction and I have used different frameworks such as deeplearning4j and theano (keras). As you may know, one of the ...
2
votes
0answers
690 views

Training batch size in relation to number of classes in a neural network

I'm using Keras on top of Theano for neural network training. What should be my batch size in relation to the number of classes? I have 560 classes and if I use a batch size more than 128, I can't ...
6
votes
1answer
206 views

Can small SGD batch size lead to faster overfitting?

I have feedforward neural net, trained on cca 34k samples and tested on 8k samples. There is 139 features in dataset. The ANN does classification between two labels, 0 and 1, so I am using sigmoid ...

15 30 50 per page