Is Training my deep learning model 100 epochs two times instead of 200 at once gonna effect the performance?

Question

I was wondering if training my Deep model two times 100 epoch each, instead of 200 epochs one time gonna effect the performance of my model or not ?

Suppose first I run my model 100 epochs and then I trained it with the initialized best weights learned in first training session of 100 epochs and running it again for 100 epochs instead of training my model once with 200 epochs

I do understand how epoch works [one epoch = one forward pass and one backward pass of all the training examples] If I have Training Set of size 1000 and batch size of 100 then it will take 10 iterations to complete one epoch in each epoch we will draw 100 instances from training set without replacement.

From my understanding I don't see any problem in Training my deep learning model 100 epochs two times instead of 200 at once

Answer 1

Yes, these two approaches can be equivalent.

The other source of differences besides the neural network weights is the optimizer state, learning-rate/momentum-schedule, extent of augmentation, progressive resizing and anything else that gets adapted during training in some manner (either adaptively or following a fixed schedule). For the 2 times 100 epoch training to be equivalent to the 200 epoch training, the same schedule of all these things to be used for all the 200 epochs - which means restoring the state of these when restarting training after an interruption. This would be an issue with many commonly used approaches (e.g. Adam, RMSProp, one-cycle policy, reducing learning rate on plateaus, progressive re-sizing etc.).

Answer 2

Under the assumption that you perform learning rate decay, training twice with 100 epochs gives likely slightly better results. The idea is that the algorithm might get stuck in a local minima and by restarting it is more likely to escape. There are multiple papers on this, e.g. ["A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation", 2018]