44
$\begingroup$

As far as I know, when adopting Stochastic Gradient Descent as learning algorithm, someone use 'epoch' for full dataset, and 'batch' for data used in a single update step, while another use 'batch' and 'minibatch' respectively, and the others use 'epoch' and 'minibatch'. This brings much confusion while discussing.

So what is the correct saying? Or they are just dialects which are all acceptable?

$\endgroup$
29
$\begingroup$
  • Epoch means one pass over the full training set
  • Batch means that you use all your data to compute the gradient during one iteration.
  • Mini-batch means you only take a subset of all your data during one iteration.
| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ What is the difference between iteration and pass? $\endgroup$ – VS. Jan 4 at 21:20
  • $\begingroup$ from practical tasks, minibatch should just be called batch, every body is calling it so $\endgroup$ – datdinhquoc Jul 15 at 9:06
  • $\begingroup$ even Keras is calling it batch_size instead of minibatch_size $\endgroup$ – datdinhquoc Jul 15 at 9:07
18
$\begingroup$

One epoch typically means your algorithm sees every training instance once. Now assuming you have $n$ training instances:

If you run batch update, every parameter update requires your algorithm see each of the $n$ training instances exactly once, i.e., every epoch your parameters are updated once.

If you run mini-batch update with batch size = $b$, every parameter update requires your algorithm see $b$ of $n$ training instances, i.e., every epoch your parameters are updated about $n/b$ times.

If you run SGD update, every parameter update requires your algorithm see 1 of $n$ training instances, i.e., every epoch your parameters are updated about $n$ times.

| cite | improve this answer | |
$\endgroup$
4
$\begingroup$

"Epoch" is usually means exposing a learning algorithm to the entire set of training data. This doesn't always make sense as we sometimes generate data.

"Batch" and "Minibatch" can be confusing.

Training examples sometimes need to be "batched" because not all data can necessarily be exposed to the algorithm at once (due to memory constraints usually).

In the context of SGD, "Minibatch" means that the gradient is calculated across the entire batch before updating weights. If you are not using a "minibatch", every training example in a "batch" updates the learning algorithm's parameters independently.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Are you sure about the last paragraph? I thought that "batched" SGD uses all of the data in an epoch to slowly compute a very precise gradient. Your last sentence sounds like a mini-batch of size 1. $\endgroup$ – Matt Krause May 4 '16 at 5:34
  • $\begingroup$ Yup, original SGD has mini-batch of size 1. I think it ultimately depends on the interpretation of the software author. Very often batch==mini-batch, without documentation ever mentioning "mini-batch". $\endgroup$ – ferrouswheel May 4 '16 at 20:36
  • $\begingroup$ Err, I guess I meant batched GD uses all of the data. I usually use batch and minibatch interchangably, but "mini-batch" when I want to point out that it's really small... $\endgroup$ – Matt Krause May 4 '16 at 21:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.