1
$\begingroup$

I have been working on a regression task for the past one year and I am stuck. My project is to simulate voxel-wise MRI data using a physics-based function and train a neural network using that data to predict a continuous number (T2 of tissue). Essentially, my training set consists solely of simulated data and my validation/test sets consist solely of real MRI voxelwise data collected from healthy volunteers. Below is the learning curve I get when training a neural network with a linear activation unit as its output:

enter image description here

Below is the histogram of predicted T2 values from this model, when I tested it on the test set consisting of real MRI data. The model seems to mainly, but not only, predict one value.

enter image description here

I then tried training the same model architecture with only real data (i.e. training/val/test sets all consist of real MRI data from healthy volunteers). I got the following learning curve and histograms: enter image description here enter image description here

The histogram looks better now as it seems to predict a range. The model also seems to converge as the validation and training losses are low and have similar values. Based on this, is it the case that my model when trained on simulated data is overfitting and I need to introduce regularization (e.g dropout)? Or is it the case that, given that the model converges for the real data, my simulated data is simply not realistic enough when compared with the real data?

Would be grateful for any advice on how to progress further! Thanks!

Edit 1: I am going to try and implement a training-val set, which would contain some data from the original training set. The training-val and training sets would be independent data sets composed of simulated MRI data. Will try plotting that as well as the original validation set containing the real data to see if that can shed insight into whether this is a data mismatch problem. Will keep you posted!

Edit 2: Here is the histogram of the real and the simulated T2 ground truth data used to test and train the model respectively. Please note the following. Firstly, the simulated data has more values at higher ground truth T2s when compared to the real ground truth distribution because the real ground truth data only consists of healthy T2 values (which are lower) whereas the simulated ground truth data consists of both healthy and diseased T2 values (which are lower and higher, respectively). Secondly, the simulated, training data consists of 48,000 samples whereas the real validation and test data consists of 1,801 samples each (so 3,602 real data samples in total). The models will later be tested on diseased real data as well, but for now I only have access to data from healthy volunteers... enter image description here

$\endgroup$
4
  • $\begingroup$ Can we see ground truth (value to be predicted) for both simulated and real-world data in a histogram too? $\endgroup$ Jan 5, 2023 at 15:58
  • $\begingroup$ since your predictions are consistently too small, I see two possibilities to consider: 1) your simulator just produces smaller values (so is unrealistic and needs to be updated) 2) there is domain shift in the feature space (so we need to employ some transfer learning tricks). the comparative histogram will inform us if it's 1. $\endgroup$ Jan 5, 2023 at 16:04
  • $\begingroup$ Hello @JohnMadden, thank you for your comment and apologies for the delay in responding. Have now added in the histogram for the real and simulated ground truth T2 data. Would be grateful for your advice based on this... $\endgroup$ Jan 9, 2023 at 8:12
  • $\begingroup$ "the real ground truth data only consists of healthy T2 values (which are lower) whereas the simulated ground truth data consists of both healthy and diseased T2 values" this is at least part of the issue; I would check out transfer learning strategies, or just focus on healthy simulated data until you get the real diseased data. $\endgroup$ Jan 9, 2023 at 14:34

1 Answer 1

0
$\begingroup$

Firstly, you do seem to be getting some learning here, so your project looks good. The RMSE looks reasonable, since it is small compared to the squared values, but $R^2$ would be a better indicator.

I see that the model never predicts values above 45. Are they missing from the training data? If so, add them. If not, you seem to need to add more, or reconsider you loss function.

The tendency to predict one value often is odd, and is likely to due to either the training data or the loss function - I suggest checking them both.

$\endgroup$
2
  • $\begingroup$ Hello, thank you for your comment and apologies for the delay in responding. I have edited my original post to include a histogram of simulated and real ground truth T2 values. The model trained using only the real data was used to check the learning curve to see if the train and val losses converge and are similar. I am using the mean_squared_error as the loss function. Could the reason why the model is predicting only one value when using simulated data be due the use of a linear activation function for the output layer? Would be really grateful for any help you can provide me with on this!! $\endgroup$ Jan 9, 2023 at 8:29
  • $\begingroup$ There's nothing wrong with using a linear function for the output layer, it is quite common. (If two consecutive layers are both linear, then the composite function must also be linear, so the second layer adds no power. But that is not an issue here.) $\endgroup$ Jan 10, 2023 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.