Does training any model from scratch require more or less updates compared to fine-tuning a pre-trained model? For cancer disease classification, I have built a network from scratch, with batch size 20, number of epochs 15, and learning rate 0.001. I have also tried a pre-trained model with fine-tuning. In this case, the batch size is 7, the number of epochs is 10, and the learning rate is 0.0001. Since the batch size of the built model is larger than the batch size of the fine-tuned model, the number of iterations per epoch is smaller, and also the total number of iterations (all epochs) is smaller. Weight updates occur after each iteration, and thus the number of updates with the built model is less than the number of updates with the fine-tuned model. The classification accuracy of the built model is slightly less than the accuracy of the pre-trained model. Could you please help me interpreting the reasons?
1 Answer
$\begingroup$
$\endgroup$
6
There is no such rule. From an optimization point of view, both are weight initializations. Whether one is better than the other depends on how your data compares to the data that the pre-trained model is trained on.
-
$\begingroup$ What about the number of epochs? with building from scratch, do we usually need more number of epochs than that required for a pre-trained model? $\endgroup$– NohaCommented Jun 3, 2022 at 11:31
-
-
$\begingroup$ According to the above link, building from scratch usually takes much longer time for training compared to pre-trained models. $\endgroup$– NohaCommented Jun 3, 2022 at 11:56
-
1$\begingroup$ I thought I had answered the question "do we usually need similar number epochs ...", which is why I said Usually, not. Sorry for the misunderstanding. Let me rectify. Usually, you need less epochs because you are assumed to choose a suitable network for your problem. That is why I say it depends. So, you can't pick any pre-trained model and expect smaller number of epochs. This decision is more critical when the network is deeper. $\endgroup$– gunesCommented Jun 3, 2022 at 12:17
-
1$\begingroup$ If they're the same data, you should be able to use your pre-trained model directly, w/o fine tuning. If it's not the same, but similar, you should expect less epochs in general, which is I believe your case. If it's not, either the network is not deep enough to notice the difference, or you may be missing some preprocessing steps that the pre-trained model applies before inputting the data, so the distribution of input features significantly differ than yours. $\endgroup$– gunesCommented Jun 3, 2022 at 12:21