Is it correct to evaluate Neural Network after a fixed number of batch-updates, rather than at the end of epoch?

Question

I'm training a neural network on a number of datasets of different size with a fixed batch size and an exponential learning decay. Normally, I would evaluate model performance, save checkpoint and reduce learning rate at the end of epoch. But in the case with multiple datasets, is correct to do these steps after a fixed number of batch-updates (say 2000)?

Answer 1

As @generic_user already written in his comment: Yes.

"Epoch" refers to a part of training during which the model is trained on every sample from the training set. It is a logical point at time to evaluate the model performance on the validation set and possibly resort to early stopping, because at this point every training sample was used same number of times.

However, there is nothing special about epochs. Especially when you are comparing your model on multiple datasets with different size, "epoch" will refer to different number of batches in each dataset, in turn meaning a different number of weight updates. Since the "distance" the weight configuration can travel in the weight space depends linearly on the number of weight updates, a model trained on a smaller dataset will manage to cover shorter distance in the weight space per epoch compared to a model trained on larger dataset.

When you are changing the learning rate after a fixed number of "steps", than these steps should be preferably batch updates rather than epochs.

Also, note that when using data augmentation scheme which can generate nearly infinitely many samples (often the case), it gets unclear what is actually an epoch, because you will be using "new" samples at every step.