I'm training on a skewed image dataset of 5000 images with class weights. The training loss decreases well and swift, but the validation-set loss fixates after around 6 epochs. I have tried using my custom model and also with transfer learning, with multiple hyperparameters. It follows the same structure everywhere. I am using categorical cross-entropy along with SGD with Nesterov momentum. This does not look like overfitting as the validation set stops performing just after 5 epochs(does not even fit).