I have two sets of data samples. Set 1 has 1900 samples and Set 2 has 1000 samples (none of which overlap with Set 1).
I am using Set 1 to train my neural network and then testing it in Set 2. On retraining my neural network the $R^2$ value that I obtained on Set 2 keeps changing. Here are the results:
Round 1. $R^2$=0.48
Round 2. $R^2$=0.6
Round 3. $R^2$= 0.98
(NOTE: In each round the hyperparameters remain same, and training is done using only set 1 where data set is split as 70-10-20 for training, validation and testing) The $R^2$ value here obtained is for set 2.
I understand that due to the random selection of training, validation and testing sets, the $R^2$ value is changing for set 2. But can I then pick Round 3 weights as the best neural network model? Or is there something wrong with my model for giving such disparate results?
Any feedback is helpful. TIA