Linked Questions

2 votes
0 answers

How to choose a good batch size for a powerful CPU [duplicate]

I am training a deep neural networks for self driving cars using Adam optimization, and I wonder how can I find a standard batch size value , currently I am using the value 1 and I can see that my ...
ob21's user avatar
  • 21
2 votes
0 answers

Choosing an ideal minibatch size for a large sample [duplicate]

I know that bigger batch size gives more accurate results, but I'm not sure which batch size is ideal given the following cases: Training on 65000 examples and validating on 13000 examples Training ...
Lucas G.'s user avatar
0 votes
0 answers

minimum data size required for good performance of mini batch gradient descent [duplicate]

I have been doing work on Theano based autoencoder. For data size of less than 100, it is working perfectly using batch gradient descent. But for data size around 500, it is better to use mini batch ...
Shyamkkhadka's user avatar
299 votes
6 answers

What is batch size in neural network?

I'm using Python Keras package for neural network. This is the link. Is batch_size equals to number of test samples? From ...
user2991243's user avatar
  • 3,951
150 votes
6 answers

Batch gradient descent versus stochastic gradient descent

Suppose we have some training set $(x_{(i)}, y_{(i)})$ for $i = 1, \dots, m$. Also suppose we run some type of supervised learning algorithm on the training set. Hypotheses are represented as $h_{\...
user20616's user avatar
  • 1,541
55 votes
1 answer

How large should the batch size be for stochastic gradient descent?

I understand that stochastic gradient descent may be used to optimize a neural network using backpropagation by updating each iteration with a different sample of the training dataset. How large ...
Simon Kuang's user avatar
  • 2,101
36 votes
4 answers

How does batch size affect convergence of SGD and why?

I've seen similar conclusion from many discussions, that as the minibatch size gets larger the convergence of SGD actually gets harder/worse, for example this paper and this answer. Also I've heard of ...
dontloo's user avatar
  • 15.9k
9 votes
2 answers

How does batch size affect Adam Optimizer?

What impact does mini-batch size have on Adam Optimizer? Is there a recommended mini-batch size when training a (covolutional) neural network with Adam Optimizer? From what I understood (I might be ...
Hello Lili's user avatar
12 votes
3 answers

Neural Networks: Is an epoch in SGD the same as an epoch in mini-batch?

In SGD an epoch would be the full presentation of the training data, and then there would be N weight updates per epoch (if there are N data examples in the training set). If we now do mini-batches ...
James's user avatar
  • 121
14 votes
1 answer

Deep Learning: Why does increase batch_size cause overfitting and how does one reduce it?

I used to train my model on my local machine, where the memory is only sufficient for 10 examples per batch. However, when I migrated my model to AWS and used a bigger GPU (Tesla K80), I could ...
infomin101's user avatar
  • 1,683
13 votes
2 answers

Minimum Training size for simple neural net

There's an old rule of thumb for multivariate statistics that recommends a minimum of 10 cases for each independent variable. But that's often where there is one parameter to fit for each variable. ...
Mike Kruger's user avatar
9 votes
2 answers

Too large batch size

I experiment with CIFA10 datasets. With my model I found that the larger the batch size, the better the model can learn the dataset. From what I see on the internet the typical size is 32 to 128, and ...
Konstantin Solomatov's user avatar
8 votes
1 answer

Can small SGD batch size lead to faster overfitting?

I have feedforward neural net, trained on cca 34k samples and tested on 8k samples. There is 139 features in dataset. The ANN does classification between two labels, 0 and 1, so I am using sigmoid ...
Jan Musil's user avatar
  • 301
6 votes
1 answer

Online learning in LSTM

Recently, I have been working on RNNs (LSTM specifically) to do time series prediction and I have used different frameworks such as deeplearning4j and theano (keras). As you may know, one of the ...
ahajib's user avatar
  • 366
5 votes
1 answer

Training batch size in relation to number of classes in a neural network

I'm using Keras on top of Theano for neural network training. What should be my batch size in relation to the number of classes? I have 560 classes and if I use a batch size more than 128, I can't ...
venuktan's user avatar
  • 151

15 30 50 per page