There are so many aspects one could possibly change in a deep neural network that it is generally not feasible to do a grid search over all of them.
True. But an alternative to that, i.e. random search is feasible in many cases. Please have a look at this post for an interesting explanation.
Hence, when tuning the various components of a neural network by hand, what is considered to be a sensible order?
The hyper-parameters interact but for practical purposes they can be tuned independently, as these interactions have no apparent structure. So, the order in which the hyper-parameters need to be tuned is largely subjective. But one of the recommendations from this paper, which I do follow is, one should tune the learning rate first. That saves a lot of experimentation. For illustration of the importance of the learning rate have a look at the image taken from the linked paper. They have experimented with different variants of LSTM over three datasets and have presented the performance over the test set. The chart shows what fraction of the test set performance variance can be attributed to different hyper-parameters
They also show that optimal value of learning rate is dependent on the dataset.
So, if I have to answer the order that I follow for training neural networks I would stick with this:
b) Learning rate
c) batch size
d) Input noise
e) Network design -number of hidden layers and number of neurons
f) Regularizers - (L1, L2, dropout etc.)
But, again, every dataset is different and hyper-parameters will surely be dependent on that. So, for every problem one approach won't do. Plotting the error will give the feel for the dataset and help in finding the 'optimal' hyper-parameters.
Some posts that might be useful:
a) In what order should we tune hyperparameters in Neural Networks?
b) Hyperparameter tuning for machine learning models.
c) A Conceptual Explanation of Bayesian Hyperparameter Optimization for Machine Learning
d) Hyper parameters tuning: Random search vs Bayesian optimization
e) hyperparameter tuning in neural networks