I am working on the development of a deep learning model for prediction of a disease from medical images. It is a binary classification algorithm. I am currently using a model built from scratch with a quite traditional architecture of several convolutional, pooling and batch normalization layers, followed by a fully connected network.
The model fits the training data well (with obvious overfitting). The loss of both the training and validation decrease over the epochs at a similar rate. The accuracy for training is close to 1 after 200 epochs, however the accuracy for validation stabilizes to approximately 0.6 after ~20 epochs. See the image below.
Any idea how to optimize? I have tried to fine-tune a lot of different hyperparameters (drop out rates, loss functions, activation functions, regularization, learning rate, et cetera) but can't seem to optimize it any further at this point.
Your input would be highly appreciated!