I want to choose the best model from a list of neural network models. My problem is a multivariate time series forecast (regression) problem, in which I forecast a parameter using other parameters, up to 4 time steps ahead.
I have read this Q&A, but in that question, no practical way is presented to select the best model from a list of models.
After reading this article, I decided to do the following procedure:
- Create different neural network architectures.
- Using the picture below, for each Neural Network (NN) architecture, perform 3 nested cross-validation (CV) . calculate RMSEs for each test data, and then calculate the average of these 3 RMSEs.
- compare These average RMSEs of different architectures. The architecture with the lowest average RMSE is my best NN model.
Train the chosen architecture with all data.
predict and see the results of the trained model on my pre-reserved test data.
The author claims that this procedure makes an unbiased error. It refers to this line of (Varma and Simon 2006)'s paper:
A nested cross-validation procedure provides an almost unbiased estimate of the true error.
To be honest, I have doubts that this nested CV is unbiased.
I have some questions:
- Is 3 nested CV enough? Maybe 5 is better?
- Does my procedure have any defections?
- How about using traditional K-Fold CV? Is it completely false?