I want to choose the best model from a list of neural network models. My problem is a multivariate time series forecast (regression) problem, in which I forecast a parameter using other parameters, up to 4 time steps ahead.

I have read this Q&A, but in that question, no practical way is presented to select the best model from a list of models.

After reading this article, I decided to do the following procedure:

  1. Create different neural network architectures.
  2. Using the picture below, for each Neural Network (NN) architecture, perform 3 nested cross-validation (CV) . calculate RMSEs for each test data, and then calculate the average of these 3 RMSEs.

enter image description here

  1. compare These average RMSEs of different architectures. The architecture with the lowest average RMSE is my best NN model.
  2. Train the chosen architecture with all data.

  3. predict and see the results of the trained model on my pre-reserved test data.

The author claims that this procedure makes an unbiased error. It refers to this line of (Varma and Simon 2006)'s paper:

A nested cross-validation procedure provides an almost unbiased estimate of the true error.

To be honest, I have doubts that this nested CV is unbiased.

I have some questions:

  • Is 3 nested CV enough? Maybe 5 is better?
  • Does my procedure have any defections?
  • How about using traditional K-Fold CV? Is it completely false?
  • $\begingroup$ Also: stats.stackexchange.com/questions/8807/… $\endgroup$
    – mkt
    May 16, 2019 at 9:17
  • $\begingroup$ @mkt, I think those question don't discuss about how to choose the best model based on the average of RMSE's and also the validity of this action itself. They just generally discuss about the data splitting. Am I correct? $\endgroup$
    – hyTuev
    May 16, 2019 at 9:25
  • $\begingroup$ You're right, my mistake. $\endgroup$
    – mkt
    May 16, 2019 at 9:36
  • $\begingroup$ I'll just leave this here as a slightly relevant link: stats.stackexchange.com/questions/14099/… $\endgroup$
    – mkt
    May 16, 2019 at 9:36
  • $\begingroup$ Hello, did you find an answer to this / what did you end up doing ? $\endgroup$
    – Perl
    Sep 7, 2019 at 15:24


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.