outer folds errors in nested cross-validation

Question

I have a time series data that I wish to be able to obtain the general performance of it. For that, I use nested cross-validation with time series flavor as described in this amazing blog.

As you have to start with an initial window size, I choose my window size to be 52. The outer fold is mainly done at each time step left, from timestep 53 onwards.

For each outer fold, I compute the average error, in this case, RMSE in order to be able to see the error's fluctuation/trend as we move through the time steps. I wish by the end of the day to be able to see that the error decreases as I move forward with timesteps and to reach a point were it stabilizes.

I know that the error in each outer fold might be affected with the choice of the hyperparameters from the inner fold. However, I do have model stability (i.e. the chosen set of hyper parameters from the inner fold is the same). However, I plot the change in the outer fold's RMSE over the time steps and this is what I got:

The so-called 'testing errors' is the RMSE of each outer fold, and the so-called 'validation errors' is the average RMSE of each inner fold.

I am surprised from the harsh fluctauations in the outer fold RMSE. Can someone explain:

• Is that normal ?
• If not, what might be the reason?
• Also if not, what shall I do to make this better and more stable ?
• The validation error is stable, which is a good sign, isn't it ?

Note: My dataset is small in size (154 samples). I am unable to collect more data.Also, the

Answer 1

If I understand correctly what you are doing,

• each of the red dots is the estimated performance by the inner crossmoving window validation. I.e., it is an RMSE based on 5 cases,
• each of the blue dots is the estimated performance by the outer crossmoving window validation, i.e. the "RMSE" based on 1 case (the next one after the 52 training cases). In this case, this RMSE over 1 case is also the absolute error for that case, which is why I put the RMSE in quotes, and
• this is done 102 times, testing cases 53 - 154:

t t t ... t t t r r r r r b x x x ... x x x 
x t t t ... t t t r r r r r b x x x ... x x 
x x t t t ... t t t r r r r r b x x x ... x 
.
:
x x x ... x x x t t t ... t t t r r r r r b 

with x = unused, t = inner training, r = red = inner testing = hyperparameter tuning, b = outer testing

Then:

• The random uncertainty of the RMSE depends on the number of cases that enter the calculation of the RMSE: the more cases, the lower the variance on the RMSE estimate, thus the less noisy we expect the estimate to look.
  So the blue estimates being based on 1 case each are expected to be more noisy than the red estimates which are based on 5 cases each.

• Unless you retrain after fixing hyperparameters on t t t ... t t t r r r r r (do you?), the blue predictions have more lag, i.e. are further into future. This may lead to higher variance error in the predictions, thus also of the blue error.

• Also, if the inner RMSE estimate is based on 5 consecutive time points and shifted one at a time, it must have considerable auto-correlation (which it does). Autocorrelation will make it look ness noisy.

• Due to the complex interaction (the inner RMSE estimte being actually part of the training) I find it difficult to say whether the outer RMSE estimate is surprisingly noisy. A simulation of the behaviour of the two RMSEs may be helpful.

---

As RMSE is the square root of the average squared error (as opposed to averaging RMSEs) the variance uncertainty of our RMSE estimate does not quite follow the simple variance of the mean $s^2_\bar x = \frac{1}{n} s^2_x$ dependency. See variance of variance estimate for more information.

---

As a side note, I had only a very quick glance at the blog post you linked, so I cannot quite comment on its total virtue. However, already the first glance did raise some red flags:

• RMSE over 1 case is just absolute error of that one case.
  Of course, one can use this - but a reminder of this property to the unsuspecting reader would have been good. I wouldn't have mentioned this, though, if there weren't the nex concern:

• Taking the average of $i$ 1-case-"RMSE"s is not the same as calculating the RMSE over $i$ cases, it's the MAE of those $i$ cases. Again, it is not forbidden to do this - but it should IMHO be clearly labeled as "mean RMSE as opposed to RMSE of the pooled predictions", and again the unsuspecting reader should be made aware of what is done. Also a justification would be in place, IMHO.

  Together, these two points make me suspicious whether the author actually calculates what they think they are calculating (to me this has a distinct "smell" of possible error with operator precedence).

• Looking at RMSE of models with a varying length of the training window makes a lot of sense. But pooling these results across widely varying training window sizes IMHO does require at the very least a justification.
  I don't have access to the papers the blog references, but I'd have expected methods with rolling origin in their name to actually have a moving origin, i.e. a moving training window of fixed size. But the day-chaining described as being the same does use expanding windows with fixed origin (and moving end).

• "cross validation" seems to me a rather unlucky choice of name as it's explicitly about not doing cross validation due to the time dependency - though to be fair that may be the result of the blog post author trading off unlucky choice of name vs. term established by the Bergmeir & Benítez paper.
  I'd have used expanding window validation (or rolling window valiation in your case) instead.

Answer 2

@cbeleites Thank you so much!!! Indeed I realize what you are saying is very very true. The reason why I did not using rolling forecasts directly is because I have suite of models: Lasso, Ridge.., etc. from which I want to pick among them the best performing model, and then do rolling forecast using THAT picked model afterwards. However I want to tune their hyper parameters. That is why I am resorting to nested cross validation. Now after your comment, I am doing the following: n the outer loop, I divide the data now into 70 training and 20 testing, and the 70 training I divide them in the inner loop into 50 training and 10 validation. As I will use expanding window as you have suggested, the training sizes in both the inner and the outer loop increase one time step at a time, with the testing and validation sizes being fixed to 20 and 10 respectively in each of the folds (inner and outer)

This is what I got so far:

Now it is very clear, thanks to you, that doing "RMSE" on 1 sample was a very big, let me call it, mistake (since as you said it is just the absolute error).