I am training CNNs for image segmentation on a limited dataset and apply some on-the-fly data augmentation. I measure mean intersection over union (mean IoU) to evaluate the training and select models. This metric increases at the beginning but it slows down strongly and almost stagnates after a while (see TensorBoard or figures below). However, it appears to continue growing (although very slowly), presumably for a long time to come. At this point (300 epochs and about 15 h into the training), it does not make the impression that we soon reach a point where clear over fitting becomes visible. When I look at validation loss, however, the point of over fitting seems to haven been passed 250 epochs ago. It should be noted, though, that loss is weighed categorical crossentropy and class frequencies are very different between training and validation data.

Where do I draw the line now? Am I underfitting the model by cutting the training at 300 epochs, or is it already overfitting? In case it is underfitting, does it make sense to wait until validation mean IoU starts to decrease, or would you expect the remaining increase of validation mean IoU negligible (since the curve becomes almost horizontal towards the end)?

mean IoU for training (orange) and validation data (blue)

Fig. 1 Mean IoU for training (orange) and validation data (blue)

Epoch loss for training (orange) and validation data (blue)

Fig. 2 Epoch loss for training (orange) and validation data (blue)

  • 2
    $\begingroup$ In both figures, it appears that the model overfits almost immediately, because there is a large disparity between the train and validation metrics. $\endgroup$
    – Sycorax
    Commented Jul 5, 2022 at 18:16
  • $\begingroup$ @Sycorax Wouldn't overfitting mean that validation metrics get worse? I mean I would not expect it something negative when the model continues to get better on the training data while not having to pay for these improvements with getting worse on the validation data? At least in case of mean IoU it looks like this is the case. I thought the large gap is simply because the training and validation data are not very similar... $\endgroup$ Commented Jul 5, 2022 at 18:42
  • $\begingroup$ @ManuelPopp overfitting boils down to not generalizing beyond training data, this is exactly what you see. If there's a discrepancy between training and evaluation data you should get better data first. $\endgroup$
    – Tim
    Commented Jul 5, 2022 at 19:17
  • $\begingroup$ If mean IoU of 0.3 is good enough to solve your problem, then there's no issue. But if you need a model with more predictive power, then these plots are a clear signal that this model isn't doing what you need. $\endgroup$
    – Sycorax
    Commented Jul 5, 2022 at 19:17
  • $\begingroup$ The problem is that I cannot get more/better data. The dataset is fix. So my question is not "where is the perfect model?" but rather "from all these bad models, which one is the best?" I expected the dataset to be insufficiently large and the differences between training and validation data to be too large. But I want to know what is the best I can make from it in order to evaluate how bad the situation is. $\endgroup$ Commented Jul 5, 2022 at 19:28


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.