Is it okay to concatenate a neural network training process repetitively using different sets of hyperparameters?
I am asking because I made a mistake when trying to optimize the hyperparameters of a vanilla NN. Instead of create a new network instance every iteration for each set of hyperparameters, I put the initiation function out of the loop (so only one instance is created).
So the initial weights for next iter training process (with a new set of hyperparameters) is using the weights trained based on last set of hyperparameters. However, doing this gives me higher validation accuracy.
My first thought was the results improved because of better/smarter weight initialization. But I still don't feel this is a sensible approach. Any ideas or comments about why and why not to do so are appreciated!