I am currently trying to train pretrained convolutional neural networks trained on the imagenet dataset to be able to classify ct-scans into two classes. Viral Pneumonia and Normal.
I am using K-fold cross validation to check if and how much my models overfit.
Specifically, I am training validating and testing my models on a ct-scan dataset and after that I also test them on an external dataset to assess their generalization ability.
After training and validating some of the pretrained models I found out that on some folds the model overfits and on other folds it is not.
The problem is that while the strategies that i have followed to reduce the problem of overfitting (data augmentation, adding dropout layers) reduce the problem on the folds that have an overfitting problem, they also cause the validation accuracy and loss to be lower than the training accuracy and loss on the folds that don't have a problem of overfitting.
Is there something i can do to lower the problem of overfitting on the folds that have it without affecting the folds that dont have a problem of overfitting?
Thank you in advance.