Neural Networks: Is an epoch in SGD the same as an epoch in mini-batch?

Question

In SGD an epoch would be the full presentation of the training data, and then there would be N weight updates per epoch (if there are N data examples in the training set).

If we now do mini-batches instead, say in batches of 20. Does one epoch now consist of N/20 weight updates, or is an epoch 'lengthened' by 20 so that it contains the same number of weight updates?

I ask this as in a couple of papers learning seems to be too quick for the number of epochs stated.

Answer 1

In the neural network terminology:

• one epoch = one forward pass and one backward pass of all the training examples
• batch size = the number of training examples in one forward/backward pass. The higher the batch size, the more memory space you'll need.
• number of iterations = number of passes, each pass using [batch size] number of examples. To be clear, one pass = one forward pass + one backward pass (we do not count the forward pass and backward pass as two different passes).

Example: if you have 1000 training examples, and your batch size is 500, then it will take 2 iterations to complete 1 epoch.

Answer 2

Franck's answer is completely correct. If you'd pay more attention to his message you'd realize you're stating the same thing as he is, just you're omitting one important aspect.

Firstly, an epoch does indeed refer to a complete pass (forward & backward) of all the training examples. This should not be a debatable topic.

Secondly, when using mini-batch gradient descent, the parameters are indeed updated at each pass. Franck and you, both stated the same thing, just differently. However, I believe your confusion originates from the fact that, when using mini-batch gradient descent (or stochastic gradient descent), multiple iterations are required to complete an epoch. Aspect clearly outlined by Franck.

For clarity, if there are n training samples, there are n / (# of mini-batches) iterations required for mini-batch GD to complete an epoch, n iterations for SGD, and evidently 1 for batch GD.

I believe that, in general, this is a confusing topic and it is important for others to have a clear understanding of the meaning of a batch, mini-batch, epoch, pass, and iteration. Thus, confusing messages like yours should be highlighted and corrected.

Answer 3

Epoch is a word that means a single pass through a training set, not all training examples.

So, yes. If we do mini-batches GD instead of a batch GD, say in batches of 20, One epoch now consist of N/20 weight updates. N is the total number of samples.

To be verbose, In a batch gradient descent, a single pass through the training allows you to take only one gradient descent step. With mini-batch (batch size = 5,000) gradient descent, a single pass through the training set, that is one epoch, allows you to take 5,000 gradient descent steps.