2
$\begingroup$

I am new to machine learning. I am looking at using time series data with a recurrent neural network in particular LSTM.

My question is to do with batch sizes. As I understand LSTM have long term memory and can work well with time series data.

I believe (please correct me if I am wrong here) that after each batch size has been passed into the model the networks parameters are updated.

So if you want to take advantage of the long term memory should you use the largest batch size possible to improve results?

In my mind if the batch size is too small during the iteration we might be missing out on some of the information.

$\endgroup$

1 Answer 1

3
$\begingroup$

Selecting a too small batch size is a problem for both LSTMs and any other neural network, not because you're "missing out on information" but because the high variance in the gradient might hamper convergence of the optimizer.

You don't take advantage of the long term memory any more or less by changing batch size.

As a rule of thumb, somewhere between 4 and 1024 is probably the optimal batch size, but you can't really tell without actually trying it out.

$\endgroup$
2
  • $\begingroup$ Then why not just choose 1024? $\endgroup$ Jan 21, 2022 at 22:34
  • 2
    $\begingroup$ @user3607022 larger batch sizes are computationally expensive $\endgroup$
    – shimao
    Jan 22, 2022 at 0:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.