Firstly I'm new to Cross Validated so I apologize if this is structured incorrectly or I didn't find some related post or missed out something.
I'm training deep networks for semantic segmentation using negative log likelihood loss (nll_loss in Pytorch). I'm evaluating every epoch using a validation set with Intersection over Union (IoU). I'm finding that the average validation loss for an epoch (averaged over batches) will eventually start increasing epoch to epoch, as the training loss reduces. I assume this is where overfitting starts. But what confuses me is that mean IoU on the validation set continues to increase after this point. Could this be to do with class imbalance? What's going on?