So there has been similar posts but none of them solves my problem, so I decided to created a new question.
I'm working on a regression project where I intend to use CNN to predict material properties, from simulation results. There are obvious relations between the input (images, containing material information) and the output (material properties) since it is the simulation that creates the results following math calculations. I have tried a deep 12-layer CNN, resnet, and VGG16 but none gave me good results. In fact, they all doing well on the training set but poorly on the testing set:
The predictions on the testing set are practically just the mean of my dataset.
I think what's happening is that, instead of learning, the models memorize the information in the training set. And I believe the solution is to somehow make it harder for the model to memorize than to learn. But the question is, how do I do this? I have tried general techniques such as image augmentation (due to problem specific reasons, the input does not allow things like distortion, rotation or cropping, so I applied only random mirroring), regularization, dropout etc., but none of these work.
I found my issue is a lot similar to this one here My Neural Net can overfit but not generalize, but there is not an answer given.
Any suggestion would be very much appreciated!