General strategy for tuning a ML algorithm

Question

I am a bit confused about the general strategy to adopt for the set up of a ML learning algorithm, in the case for example of a neural network, and I have not been able to find resources (books or articles) to answer my questions.

Let us consider a neural network where the variables that are tunable are the numbers of layers, the sizes of the layers and the connections between layers (for example relu, sigmoid, pooling, convolutions...) and that I have a training set, a validation set and a test set.

From what I have understood about the subject, a general strategy could be the following: I choose a neural network architecture (as described above) and I train my neural network, in the sense that with an iterative method (like gradient descent), weights (biases included) are iteratively computed in order to minimize the discrepancy (in some given norm) between the actual values $y_{train}$ and the output given by the neural network $\hat y_{train}$. Then, at each iteration, I use the validation set to compute the discrepancy between the actual values $y_{val}$ and the output given by the neural network $\hat y_{val}$.

So my questions are:

1) Should I tune the parameters (number of layers, sizes of layers, type of connections) in order to minimize the error between $y_{val}$ and $\hat y_{val}$ ?

2) Isn't doing 1) considered data snooping ?

3) If I pursue the strategy described in 1), suppose I obtain after tuning the neural network a very low error between $y_{val}$ and $\hat y_{val}$. Then I use the test set to compute the error between $y_{test}$ and $\hat y_{test}$ and find out that the error is larger than the validation error.

So what should I do ? Change the parameters of the neural network ? I suppose that would be again data snooping and/or that it would limit the further ability of the neural network to make correct predictions of new data ? And how much larger with respect to the validation set error is the test set error considered too large ?

4) More generally, could someone describe the strategy that is generally accepted in the ML/DL community to tune the parameters of a neural network in the case described above (in the sense where I can tune the number of layers, the sizes of layers, the type of connections and I have training, validation and test sets) ?

Answer 1

1) Should I tune the parameters (number of layers, sizes of layers, type of connections) in order to minimize the error between $y_{val}$ and $\hat y_{val}$ ?

Yes, all hyperparameters (including those governing network architecture) can be tuned using the validation set.

2) Isn't doing 1) considered data snooping ?

No, this isn't snooping/peeking because the final performance should be measured using an independent test set (distinct from the training and validation sets).

3) If I pursue the strategy described in 1), suppose I obtain after tuning the neural network a very low error between $y_{val}$ and $\hat y_{val}$. Then I use the test set to compute the error between $y_{test}$ and $\hat y_{test}$ and find out that the error is larger than the validation error. So what should I do ? Change the parameters of the neural network ?

No, you should never perform any fitting or model selection using the test set. This would constitute peeking, and would give an overoptimistic estimate of performance. Keep the test set under lock and key, and don't let it influence the model in any way (including deciding to try another method until you're happy with it).

The test set error might be larger than the validation set error. If that's the case, then so be it; this is an honest estimate of the performance of your entire learning algorithm (which includes model selection/hyperparameter tuning). If the test set error is larger, it might simply reflect random fluctuations in the dataset, or it could be that you're overfitting the validation set. If you suspect the latter might be an issue, you can use nested cross validation to make more efficient use of your data (and/or gather more data; training large networks can require very large datasets). A simpler network and/or more restricted tuning procedure could also help if it's possible to make sensible choices. As above, any decisions along these lines should not be made by looking at the test set.

4) More generally, could someone describe the strategy that is generally accepted in the ML/DL community to tune the parameters of a neural network in the case described above (in the sense where I can tune the number of layers, the sizes of layers, the type of connections and I have training, validation and test sets) ?

It sounds like you already have the gist of this. For large datasets, split the data into fixed training/validation/test sets. Fit weights using the training set, tune hyperparameters using the validation set, and estimate performance using the test set. For smaller datasets, use nested cross validation instead of fixed training/validation/test sets. This will give improved estimates at the cost of more computation.