I have two sets of data samples. Set 1 has 1900 samples and Set 2 has 1000 samples (none of which overlap with Set 1).

I am using Set 1 to train my neural network and then testing it in Set 2. On retraining my neural network the $R^2$ value that I obtained on Set 2 keeps changing. Here are the results:

Round 1. $R^2$=0.48

Round 2. $R^2$=0.6

Round 3. $R^2$= 0.98

(NOTE: In each round the hyperparameters remain same, and training is done using only set 1 where data set is split as 70-10-20 for training, validation and testing) The $R^2$ value here obtained is for set 2.

I understand that due to the random selection of training, validation and testing sets, the $R^2$ value is changing for set 2. But can I then pick Round 3 weights as the best neural network model? Or is there something wrong with my model for giving such disparate results?

Any feedback is helpful. TIA

  • $\begingroup$ What is R? Do you mean $R^2$? $\endgroup$ – Stephan Kolassa May 3 '19 at 9:18
  • $\begingroup$ Yes, I meant R2. Made the edit. Thanks! $\endgroup$ – user110565 May 3 '19 at 9:27

It's very good that you look at variability in results.

Your $R^2$ value varies a lot. Yes, as you write, this is due to random selection of training, test and validation sets. (There may also be some randomness in your NN training, depending on your specific architecture and implementation.)

No, you can't just pick the model trained in Round 3 and expect it to perform better than the others. Except if you believe that the data it will actually be applied to will be very similar to set 2. However, I would assume that both sets 1 and 2 are similar to the actual production environment in which your net will be applied (otherwise, why train on them?), so considering how variable your $R^2$ is, the only thing we can more or less reasonably say is that results in production will also vary widely. They will depend quite as much on the actual instances you see in production as they did on the split in training.

The large variation in $R^2$ looks like overfitting to me. Any but the simplest network will have a large variance on just 1900 training samples. Consider a simpler architecture, or regularization, or collecting (much) more data.

  • $\begingroup$ Unless I misunderstand, the data from Set 2 is never actually seen during training, so overfitting wouldn't be an appropriate conclusion? So if you did see a value of .98 for the R2 on the test data, that sounds like a good predictor of generalization quality to me... $\endgroup$ – bibliolytic May 3 '19 at 13:08
  • $\begingroup$ @bibliolytic: I should have been more precise. I mean "overfitting" here in the sense of "overfitting to the test set". A thought experiment: fit the model not 3 times, but 10,000 times with different randomizations. Do you believe the one that performs best on Set 2 will reliably yield the best predictions in new data? I don't think so. The model just appears to have a very high variance to me. $\endgroup$ – Stephan Kolassa May 3 '19 at 13:42
  • $\begingroup$ Given that he is using a non-convex optimization, I'm not sure if it's reasonable to assume overfitting to the test set after just 3 runs, though I certainly agree with you if he had done many thousands. It could simply be that the network failed to find a good optimum twice and managed once $\endgroup$ – bibliolytic May 7 '19 at 14:56

Because there is some variance in our opinions, I wanted to add a second response here:

Depending very much on the way you train your network, and the parameters of said network, a large variance in $R^2$ is not surprising -- and if you are confident that your test set is enough data to describe the domain of your problem, then I think you have good evidence to say that the model with an $R^2$ of 0.98 will perform well on new data. Overall, however, the information you've provided is not sufficient to make a clear statement one way or the other.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.