How to resume training in neural networks properly?

Question

I'm working on training a network to identify different kinds of cells. For each experimental batch, I would take my previous model weight, and then train a few new pictures on it. Since the model is for the same kind of data, thus this is not a transfer study in my mind. My question is that does the newly trained model pertain the old training memory, for example, if I predict an image from an older batch, would this new model perform badly comparing to the older model? And why?

Answer 1

take my previous model weight, and then train a few new pictures on

This will make the new-model a type of transfer learning practice as the weight initialisation has a prior from previous batch, despite the domain of interest is identical. Because only weight information is passed to the new-network and it will start from "fresh-training".

does the newly trained model pertain the old training memory,

Yes, due to weight initialisation.

would this new model perform badly comparing to the older model? And why?

Not necessarily but performance could degrade after repeating multiple batches. It could forget earlier samples. The reason is not that well known but phenomenon is called catastrophic forgetting. To avoid this, one should use checkpointing to resume training instead of simple weight transfer, see for example PyTorch's Saving and loading a general checkpoint model for inference or resuming training .