121 views

### How to choose a good batch size for a powerful CPU [duplicate]

I am training a deep neural networks for self driving cars using Adam optimization, and I wonder how can I find a standard batch size value , currently I am using the value 1 and I can see that my ...
71 views

### minimum data size required for good performance of mini batch gradient descent [duplicate]

I have been doing work on Theano based autoencoder. For data size of less than 100, it is working perfectly using batch gradient descent. But for data size around 500, it is better to use mini batch ...
55 views

### Choosing an ideal minibatch size for a large sample [duplicate]

I know that bigger batch size gives more accurate results, but I'm not sure which batch size is ideal given the following cases: Training on 65000 examples and validating on 13000 examples Training ...
365k views

### What is batch size in neural network?

I'm using Python Keras package for neural network. This is the link. Is batch_size equals to number of test samples? From ...
120k views

Suppose we have some training set $(x_{(i)}, y_{(i)})$ for $i = 1, \dots, m$. Also suppose we run some type of supervised learning algorithm on the training set. Hypotheses are represented as \$h_{\...
75k views

### How large should the batch size be for stochastic gradient descent?

I understand that stochastic gradient descent may be used to optimize a neural network using backpropagation by updating each iteration with a different sample of the training dataset. How large ...
32k views

### How does batch size affect convergence of SGD and why?

I've seen similar conclusion from many discussions, that as the minibatch size gets larger the convergence of SGD actually gets harder/worse, for example this paper and this answer. Also I've heard of ...
14k views

### How does batch size affect Adam Optimizer?

What impact does mini-batch size have on Adam Optimizer? Is there a recommended mini-batch size when training a (covolutional) neural network with Adam Optimizer? From what I understood (I might be ...
3k views

### Neural Networks: Is an epoch in SGD the same as an epoch in mini-batch?

In SGD an epoch would be the full presentation of the training data, and then there would be N weight updates per epoch (if there are N data examples in the training set). If we now do mini-batches ...
5k views

### Too large batch size

I experiment with CIFA10 datasets. With my model I found that the larger the batch size, the better the model can learn the dataset. From what I see on the internet the typical size is 32 to 128, and ...
3k views

### Minimum Training size for simple neural net

There's an old rule of thumb for multivariate statistics that recommends a minimum of 10 cases for each independent variable. But that's often where there is one parameter to fit for each variable. ...
9k views

### Deep Learning: Why does increase batch_size cause overfitting and how does one reduce it?

I used to train my model on my local machine, where the memory is only sufficient for 10 examples per batch. However, when I migrated my model to AWS and used a bigger GPU (Tesla K80), I could ...
3k views

### Online learning in LSTM

Recently, I have been working on RNNs (LSTM specifically) to do time series prediction and I have used different frameworks such as deeplearning4j and theano (keras). As you may know, one of the ...