I have been working on a regression task for the past one year and I am stuck. My project is to simulate voxel-wise MRI data using a physics-based function and train a neural network using that data to predict a continuous number (T2 of tissue). Essentially, my training set consists solely of simulated data and my validation/test sets consist solely of real MRI voxelwise data collected from healthy volunteers. Below is the learning curve I get when training a neural network with a linear activation unit as its output:
Below is the histogram of predicted T2 values from this model, when I tested it on the test set consisting of real MRI data. The model seems to mainly, but not only, predict one value.
I then tried training the same model architecture with only real data (i.e. training/val/test sets all consist of real MRI data from healthy volunteers). I got the following learning curve and histograms:
The histogram looks better now as it seems to predict a range. The model also seems to converge as the validation and training losses are low and have similar values. Based on this, is it the case that my model when trained on simulated data is overfitting and I need to introduce regularization (e.g dropout)? Or is it the case that, given that the model converges for the real data, my simulated data is simply not realistic enough when compared with the real data?
Would be grateful for any advice on how to progress further! Thanks!
Edit 1: I am going to try and implement a training-val set, which would contain some data from the original training set. The training-val and training sets would be independent data sets composed of simulated MRI data. Will try plotting that as well as the original validation set containing the real data to see if that can shed insight into whether this is a data mismatch problem. Will keep you posted!
Edit 2: Here is the histogram of the real and the simulated T2 ground truth data used to test and train the model respectively. Please note the following. Firstly, the simulated data has more values at higher ground truth T2s when compared to the real ground truth distribution because the real ground truth data only consists of healthy T2 values (which are lower) whereas the simulated ground truth data consists of both healthy and diseased T2 values (which are lower and higher, respectively). Secondly, the simulated, training data consists of 48,000 samples whereas the real validation and test data consists of 1,801 samples each (so 3,602 real data samples in total). The models will later be tested on diseased real data as well, but for now I only have access to data from healthy volunteers...