Random initialization of weights

Question

I have trained a neural network using a train and a validation dataset.I used the validation dataset for hyperparameter and architecture optimization.I split the dataset in the exact same way each time so there is nothing stochastic there.
However each time i train the model because of the random weight initialization i get different results on the test set.Thus i assume my model has high variance.
What i did was run many identical models(with optimal hyperparameters found with validation set) and i chose the one that has the lowest rmse loss(this is my metric) in my test dataset.

My concern is this:
Since the optimal model has a lot of variance by running 20 or 30 identical models and choosing the one with the best loss for my test set am i overfitting the model to my test set?

Answer 1

Yes, you are overfitting to the test set. There is a recent paper discussing this topic.

Ideally, you'd find a way of regularizing the training so its outcome is less variable. Or you could train lots of models and average their predictions.

If you can't find a way of getting around the problem, it is not really appropriate to report the best performing result without mentioning all the other runs, instead on ought to report the distribution of results (e.g. mean and standard deviation of performance metric).

Answer 2

You can use one of the weight initialization strategies to initialize your weights and store the weights of the best performing network and reuse them time and again to get better results.