50
$\begingroup$

As far as I know, when adopting Stochastic Gradient Descent as learning algorithm, someone use 'epoch' for full dataset, and 'batch' for data used in a single update step, while another use 'batch' and 'minibatch' respectively, and the others use 'epoch' and 'minibatch'. This brings much confusion while discussing.

So what is the correct saying? Or they are just dialects which are all acceptable?

$\endgroup$
34
$\begingroup$
  • Epoch means one pass over the full training set
  • Batch means that you use all your data to compute the gradient during one iteration.
  • Mini-batch means you only take a subset of all your data during one iteration.
$\endgroup$
3
  • 3
    $\begingroup$ What is the difference between iteration and pass? $\endgroup$ – VS. Jan 4 '20 at 21:20
  • 1
    $\begingroup$ from practical tasks, minibatch should just be called batch, every body is calling it so $\endgroup$ – datdinhquoc Jul 15 '20 at 9:06
  • 1
    $\begingroup$ even Keras is calling it batch_size instead of minibatch_size $\endgroup$ – datdinhquoc Jul 15 '20 at 9:07
22
$\begingroup$

One epoch typically means your algorithm sees every training instance once. Now assuming you have $n$ training instances:

If you run batch update, every parameter update requires your algorithm see each of the $n$ training instances exactly once, i.e., every epoch your parameters are updated once.

If you run mini-batch update with batch size = $b$, every parameter update requires your algorithm see $b$ of $n$ training instances, i.e., every epoch your parameters are updated about $n/b$ times.

If you run SGD update, every parameter update requires your algorithm see 1 of $n$ training instances, i.e., every epoch your parameters are updated about $n$ times.

$\endgroup$
5
$\begingroup$

"Epoch" is usually means exposing a learning algorithm to the entire set of training data. This doesn't always make sense as we sometimes generate data.

"Batch" and "Minibatch" can be confusing.

Training examples sometimes need to be "batched" because not all data can necessarily be exposed to the algorithm at once (due to memory constraints usually).

In the context of SGD, "Minibatch" means that the gradient is calculated across the entire batch before updating weights. If you are not using a "minibatch", every training example in a "batch" updates the learning algorithm's parameters independently.

$\endgroup$
3
  • 1
    $\begingroup$ Are you sure about the last paragraph? I thought that "batched" SGD uses all of the data in an epoch to slowly compute a very precise gradient. Your last sentence sounds like a mini-batch of size 1. $\endgroup$ – Matt Krause May 4 '16 at 5:34
  • $\begingroup$ Yup, original SGD has mini-batch of size 1. I think it ultimately depends on the interpretation of the software author. Very often batch==mini-batch, without documentation ever mentioning "mini-batch". $\endgroup$ – ferrouswheel May 4 '16 at 20:36
  • $\begingroup$ Err, I guess I meant batched GD uses all of the data. I usually use batch and minibatch interchangably, but "mini-batch" when I want to point out that it's really small... $\endgroup$ – Matt Krause May 4 '16 at 21:02
0
$\begingroup$

An epoch is typically one loop over the entire dataset.

A batch or minibatch refers to equally sized subsets of the dataset over which the gradient is calculated and weights updated.

i.e. for a dataset of size $n$:

Optimization method Samples in each gradient calculation Weight updates per epoch
Batch Gradient Descent the entire dataset $1$
Minibatch Gradient Descent consecutive subsets of the dataset $\frac{n}{\text{size of minibatch}}$
Stochastic Gradient Descent* each sample of the dataset $n$

The term batch itself is ambiguous however and can refer to either batch gradient descent or the size of a minibatch.


* Equivalent to minibatch with a batch-size of 1.


Why use minibatches?

It may be infeasible (due to memory/computational constraints) to calculate the gradient over the entire dataset, so smaller minibatches (as opposed to a single batch) may be used instead. At its extreme one can recalculate the gradient over each individual sample in the dataset.

If you perform this iteratively (i.e. re-calculate over the entire dataset multiple times), each iteration is referred to as an epoch.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.