0
$\begingroup$

So there has been similar posts but none of them solves my problem, so I decided to created a new question.

I'm working on a regression project where I intend to use CNN to predict material properties, from simulation results. There are obvious relations between the input (images, containing material information) and the output (material properties) since it is the simulation that creates the results following math calculations. I have tried a deep 12-layer CNN, resnet, and VGG16 but none gave me good results. In fact, they all doing well on the training set but poorly on the testing set:

enter image description here enter image description here

The predictions on the testing set are practically just the mean of my dataset.

I think what's happening is that, instead of learning, the models memorize the information in the training set. And I believe the solution is to somehow make it harder for the model to memorize than to learn. But the question is, how do I do this? I have tried general techniques such as image augmentation (due to problem specific reasons, the input does not allow things like distortion, rotation or cropping, so I applied only random mirroring), regularization, dropout etc., but none of these work.

I found my issue is a lot similar to this one here My Neural Net can overfit but not generalize, but there is not an answer given.

Any suggestion would be very much appreciated!

Improve this question
$\endgroup$
5
  • $\begingroup$ Thank you @mhdadk for the comment. I have read this post before but unfortunately it's not really helpful. Most importantly there is not a clear answer to it either $\endgroup$
    – xshang
    May 11 at 21:58
  • $\begingroup$ Do you mean the predictions on the test set are practically just the mean of my dataset? Looking at your plots, your training predictions seem to span the full range of target values. Have you tried using smaller models (shallower and/or narrower)? Generally smaller models will overfit less than large models. A 12-layer CNN seems quite large, unless you have a lot of training data. $\endgroup$
    – Lynn
    May 13 at 8:31
  • 2
    $\begingroup$ “Overfitting” and “not generalizing” are synonyms. $\endgroup$
    – Sycorax ♦
    May 13 at 10:45
  • $\begingroup$ Hi @Lynn, yes I meant to say one the 'test' set. Thank you for catching that. I did try with shallower networks and the results on the testing set are pretty much the same, only that on the training set it become similar to what I'm getting now on the testing set. $\endgroup$
    – xshang
    May 13 at 22:23
  • $\begingroup$ @Sycorax You are absolutely right. I guess what I was trying to say is that the model is not really learning. I'll change the wording. $\endgroup$
    – xshang
    May 13 at 22:26

0

Reset to default

Browse other questions tagged or ask your own question.