I'm building a convolutional neural network (CNN) to predict the response factor (continuous variable) of organic molecules. As input, I use 86x86 onehot-encoded matrices that represent the compounds in my dataset. Since I do not have a huge dataset, I use 80/20 training/validation split, without a separate test set. Network architecture is similar to that of VGG and a similar architecture has been successfully deployed to deal with a very similar task as described here: https://doi.org/10.1021/acs.analchem.8b05821. Shortly, it consists of a series of 2D convolutional layers [3x3 filter size, stride 1, followed by ReLu and max pooling)], global pooling and several fully connected layers. MSE is used as loss function as a MATLAB default.
Problem at hand
What has been done:
I've tried changing the configuration of the network by varying the number of convolutional and fully connected layers, applying dropout layers in front of a few fully connected layers (slightly helped), adding batch normalisation layers after convolutional layers (didn't help at all, it made performance even worse, surprisingly) and experimenting with different minibatch sizes and learning rates. It seems my network is very sensitive to initial learning rate since even minor changes can deteriorate the predictive performance. Repartitioning the dataset does not induce major changes in the shape of the loss curves or the predictive performance.
I believe I'm facing of some kind of overfitting, judging by the almost perfect predictions on the training set and worse performance on the validation set. However, in that case I'd expect the validation loss curve to take the "typical" U-shape, which doesn't happen in my case and makes me puzzled. I'm yet to give more tries to data augmentation, although the first experiments with that resulted in marginal improvement only.
What can be the explanation to this odd behaviour? Why does my network perform so badly on my validation set while the loss is approximately the same as that of the training set?