0
$\begingroup$

I am working on 3D medical image segmentation area. It may take 2-3 days to finish one round of training. How can I interpret the learning curve if over-fitting is happening or not? It happens to me that learning curve shows that loss is decreasing but in fact the model over-fits on data.

I have problem of knowing and detecting it in advance.

I would really appreciate if there is any helpful link, please share here.

Your help is really appreciated.

Improve this question
$\endgroup$
  • $\begingroup$ Do you have a test set that you can use to test your model at intermediate time points? $\endgroup$ – user2974951 Oct 15 '18 at 8:32
  • $\begingroup$ @user2974951 Since the data is 3D, it is impossible to have the test phase during training. GPU memory is limited. $\endgroup$ – sc241 Oct 15 '18 at 8:34
  • 1
    $\begingroup$ @InfProbSciX your comment pretty much answers the question. If you post it as an answer, I'll gladly upvote! $\endgroup$ – Jan Kukacka Oct 15 '18 at 12:09
1
$\begingroup$

I think what @user2974951 is suggesting is that, you could split up one massive training phase into smaller chunks. For example, instead of runnings through 10 epochs all at once, you could train your model for 2 epochs, get the parameters/weights, (save them and) test using these points. Then, you could use the pre-trained weights as initial values and start training again - hence accomplishing the test phase at intermediate points (no resources would be spent on testing during training).

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ Also make sure to save/reload the parameters of the optimizer (such as the learning rate). $\endgroup$ – Jan Kukacka Oct 15 '18 at 12:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.