I have found a set of optimal hyperparameters (e.g. learning rate for gradient descent) using cross validation and bayesian optimisation. While searching for the optimal hyperparameters, my neural net architecture remained constant (same number of layers, same number of nodes etc.).
I chose a relatively small architecture with 2 hidden layers so that the model would train and evaluate faster.
Now that I've found the optimal hyperparameters, I am wondering if I increase the number of hidden layers and nodes per layer, will the hyperparameters still be optimal? All else will remain the same (same training data and validation data).
The reason for making the network deeper and wider now, is that this will serve as the final model, which I will allow to train for more epochs to obtain the highest possible accuracy; I don't mind if it takes a few days to train 1 model now, whereas in optimising the hyperparameters I needed to train a model within a few hours.