I have trained a neural network using a train and a validation dataset.I used the validation dataset for hyperparameter and architecture optimization.I split the dataset in the exact same way each time so there is nothing stochastic there.
However each time i run the model because of the random weight initialization i get different results.Thus i assume my model has high variance.
What i did was run many identical models(with optimal hyperparameters found with validation set) and i chose the one that has the lowest rmse loss(this is my metric) in my test dataset.

My concern is this:
Since the optimal model has a lot of variance by running 20 or 30 identical models and choosing the one with the best loss for my test set am i overfitting the model to my test set?

I have trained a neural network using a train and a validation dataset.I used the validation dataset for hyperparameter and architecture optimization.I split the dataset in the exact same way each time so there is nothing stochastic there.
However each time i train the model because of the random weight initialization i get different results on the test set.Thus i assume my model has high variance.
What i did was run many identical models(with optimal hyperparameters found with validation set) and i chose the one that has the lowest rmse loss(this is my metric) in my test dataset.

My concern is this:
Since the optimal model has a lot of variance by running 20 or 30 identical models and choosing the one with the best loss for my test set am i overfitting the model to my test set?