What are the differences between 'epoch', 'batch', and 'minibatch'? As far as I know, when adopting Stochastic Gradient Descent as learning algorithm,
someone use 'epoch' for full dataset, and 'batch' for data used in a single update step, while another use 'batch' and 'minibatch' respectively, and the others use 'epoch' and 'minibatch'. This brings much confusion while discussing.
So what is the correct saying? Or they are just dialects which are all acceptable?
 A: "Epoch" is usually means exposing a learning algorithm to the entire set of training data. This doesn't always make sense as we sometimes generate data.
"Batch" and "Minibatch" can be confusing.
Training examples sometimes need to be "batched" because not all data can necessarily be exposed to the algorithm at once (due to memory constraints usually).
In the context of SGD, "Minibatch" means that the gradient is calculated across the entire batch before updating weights. If you are not using a "minibatch", every training example in a "batch" updates the learning algorithm's parameters independently.
A: *

*Epoch means one pass over the full training set

*Batch means that you use all your data to compute the gradient during one iteration.

*Mini-batch means you only take a subset of all your data during one iteration.

A: One epoch typically means your algorithm sees every training instance once.  Now assuming you have $n$ training instances:
If you run batch update, every parameter update requires your algorithm see each of the $n$ training instances exactly once, i.e., every epoch your parameters are updated once.
If you run mini-batch update with batch size = $b$, every parameter update requires your algorithm see $b$ of $n$ training instances, i.e., every epoch your parameters are updated about $n/b$ times.
If you run SGD update, every parameter update requires your algorithm see 1 of $n$ training instances, i.e., every epoch your parameters are updated about $n$ times.
A: An epoch is typically one loop over the entire dataset.
A batch or minibatch refers to equally sized subsets of the dataset over which the gradient is calculated and weights updated.
i.e. for a dataset of size $n$:




Optimization method
Samples in each gradient calculation
Weight updates per epoch




Batch Gradient Descent
the entire dataset
$1$


Minibatch Gradient Descent
consecutive subsets of the dataset
$\lceil\frac{n}{\text{size of minibatch}}\rceil$


Stochastic Gradient Descent*
each sample of the dataset
$n$




The term batch itself is ambiguous however and can refer to either batch gradient descent or the size of a minibatch.


* Equivalent to minibatch with a batch-size of 1.

Why use minibatches?
It may be infeasible (due to memory/computational constraints) to calculate the gradient over the entire dataset, so smaller minibatches (as opposed to a single batch) may be used instead. At its extreme one can recalculate the gradient over each individual sample in the dataset.
If you perform this iteratively (i.e. re-calculate over the entire dataset multiple times), each iteration is referred to as an epoch.

