I'm training a neural network on a number of datasets of different size with a fixed batch size and an exponential learning decay. Normally, I would evaluate model performance, save checkpoint and reduce learning rate at the end of epoch. But in the case with multiple datasets, is correct to do these steps after a fixed number of batch-updates (say 2000)?

  • $\begingroup$ Sure, why not? There is nothing magical about epoching, and I'd not be surprised if it's not even optimal to organize batches by sampling without replacement. $\endgroup$ Feb 19 '18 at 18:00

As @generic_user already written in his comment: Yes.

"Epoch" refers to a part of training during which the model is trained on every sample from the training set. It is a logical point at time to evaluate the model performance on the validation set and possibly resort to early stopping, because at this point every training sample was used same number of times.

However, there is nothing special about epochs. Especially when you are comparing your model on multiple datasets with different size, "epoch" will refer to different number of batches in each dataset, in turn meaning a different number of weight updates. Since the "distance" the weight configuration can travel in the weight space depends linearly on the number of weight updates, a model trained on a smaller dataset will manage to cover shorter distance in the weight space per epoch compared to a model trained on larger dataset.

When you are changing the learning rate after a fixed number of "steps", than these steps should be preferably batch updates rather than epochs.

Also, note that when using data augmentation scheme which can generate nearly infinitely many samples (often the case), it gets unclear what is actually an epoch, because you will be using "new" samples at every step.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.