As far as I know, when adopting Stochastic Gradient Descent as learning algorithm, someone use 'epoch' for full dataset, and 'batch' for data used in a single update step, while another use 'batch' and 'minibatch' respectively, and the others use 'epoch' and 'minibatch'. This brings much confusion while discussing.

So what is the correct saying? Or they are just dialects which are all acceptable?

  • 1
    $\begingroup$ If any of the responses answer your question you might consider marking it "accepted". $\endgroup$ Commented Aug 22, 2021 at 13:13

4 Answers 4

  • Epoch means one pass over the full training set
  • Batch means that you use all your data to compute the gradient during one iteration.
  • Mini-batch means you only take a subset of all your data during one iteration.
  • 5
    $\begingroup$ What is the difference between iteration and pass? $\endgroup$
    – VS.
    Commented Jan 4, 2020 at 21:20
  • 4
    $\begingroup$ from practical tasks, minibatch should just be called batch, every body is calling it so $\endgroup$
    – Dan D.
    Commented Jul 15, 2020 at 9:06
  • 2
    $\begingroup$ even Keras is calling it batch_size instead of minibatch_size $\endgroup$
    – Dan D.
    Commented Jul 15, 2020 at 9:07
  • $\begingroup$ Pass is one computation of gradients and one update of weights. Iteration is a run of a single batch, that theoretically can have multiple passes. Someone can update weights multiple times while processing one randomly selected batch: you'll have multiple passes but single iteration. In most cases one pass is one iteration. $\endgroup$ Commented Sep 19, 2023 at 20:41

One epoch typically means your algorithm sees every training instance once. Now assuming you have $n$ training instances:

If you run batch update, every parameter update requires your algorithm see each of the $n$ training instances exactly once, i.e., every epoch your parameters are updated once.

If you run mini-batch update with batch size = $b$, every parameter update requires your algorithm see $b$ of $n$ training instances, i.e., every epoch your parameters are updated about $n/b$ times.

If you run SGD update, every parameter update requires your algorithm see 1 of $n$ training instances, i.e., every epoch your parameters are updated about $n$ times.


An epoch is typically one loop over the entire dataset.

A batch or minibatch refers to equally sized subsets of the dataset over which the gradient is calculated and weights updated.

i.e. for a dataset of size $n$:

Optimization method Samples in each gradient calculation Weight updates per epoch
Batch Gradient Descent the entire dataset $1$
Minibatch Gradient Descent consecutive subsets of the dataset $\lceil\frac{n}{\text{size of minibatch}}\rceil$
Stochastic Gradient Descent* each sample of the dataset $n$

The term batch itself is ambiguous however and can refer to either batch gradient descent or the size of a minibatch.

* Equivalent to minibatch with a batch-size of 1.

Why use minibatches?

It may be infeasible (due to memory/computational constraints) to calculate the gradient over the entire dataset, so smaller minibatches (as opposed to a single batch) may be used instead. At its extreme one can recalculate the gradient over each individual sample in the dataset.

If you perform this iteratively (i.e. re-calculate over the entire dataset multiple times), each iteration is referred to as an epoch.

  • 1
    $\begingroup$ love the use of a table $\endgroup$
    – jojorabbit
    Commented Nov 14, 2022 at 10:49

"Epoch" is usually means exposing a learning algorithm to the entire set of training data. This doesn't always make sense as we sometimes generate data.

"Batch" and "Minibatch" can be confusing.

Training examples sometimes need to be "batched" because not all data can necessarily be exposed to the algorithm at once (due to memory constraints usually).

In the context of SGD, "Minibatch" means that the gradient is calculated across the entire batch before updating weights. If you are not using a "minibatch", every training example in a "batch" updates the learning algorithm's parameters independently.

  • 1
    $\begingroup$ Are you sure about the last paragraph? I thought that "batched" SGD uses all of the data in an epoch to slowly compute a very precise gradient. Your last sentence sounds like a mini-batch of size 1. $\endgroup$ Commented May 4, 2016 at 5:34
  • $\begingroup$ Yup, original SGD has mini-batch of size 1. I think it ultimately depends on the interpretation of the software author. Very often batch==mini-batch, without documentation ever mentioning "mini-batch". $\endgroup$ Commented May 4, 2016 at 20:36
  • $\begingroup$ Err, I guess I meant batched GD uses all of the data. I usually use batch and minibatch interchangably, but "mini-batch" when I want to point out that it's really small... $\endgroup$ Commented May 4, 2016 at 21:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.