# outer folds errors in nested cross-validation

I have a time series data that I wish to be able to obtain the general performance of it. For that, I use nested cross-validation with time series flavor as described in this amazing blog.

As you have to start with an initial window size, I choose my window size to be 52. The outer fold is mainly done at each time step left, from timestep 53 onwards.

For each outer fold, I compute the average error, in this case, RMSE in order to be able to see the error's fluctuation/trend as we move through the time steps. I wish by the end of the day to be able to see that the error decreases as I move forward with timesteps and to reach a point were it stabilizes.

I know that the error in each outer fold might be affected with the choice of the hyperparameters from the inner fold. However, I do have model stability (i.e. the chosen set of hyper parameters from the inner fold is the same). However, I plot the change in the outer fold's RMSE over the time steps and this is what I got:

The so-called 'testing errors' is the RMSE of each outer fold, and the so-called 'validation errors' is the average RMSE of each inner fold.

I am surprised from the harsh fluctauations in the outer fold RMSE. Can someone explain:

• Is that normal ?
• If not, what might be the reason?
• Also if not, what shall I do to make this better and more stable ?
• The validation error is stable, which is a good sign, isn't it ?

Note: My dataset is small in size (154 samples). I am unable to collect more data.Also, the

• what is the size of training and validation sets? – cbeleites unhappy with SX Sep 16 '19 at 14:12
• @cbeleites Thank you so much for your reply. In the outer fold, I divided the data into training size of 52 and test size of 1 (The choice of 1 is because I am dealing with time series, and I am aiming to do single step forecasts, which implicates a test size of 1). In the inner fold, the training data (of size 52) is divided further into 47 training and 5 testing. If I use a sliding window, these sizes keep the same, I only shift my samples as I go forward. – Perl Sep 17 '19 at 11:29

If I understand correctly what you are doing,

• each of the red dots is the estimated performance by the inner crossmoving window validation. I.e., it is an RMSE based on 5 cases,
• each of the blue dots is the estimated performance by the outer crossmoving window validation, i.e. the "RMSE" based on 1 case (the next one after the 52 training cases). In this case, this RMSE over 1 case is also the absolute error for that case, which is why I put the RMSE in quotes, and
• this is done 102 times, testing cases 53 - 154:
t t t ... t t t r r r r r b x x x ... x x x
x t t t ... t t t r r r r r b x x x ... x x
x x t t t ... t t t r r r r r b x x x ... x
.
:
x x x ... x x x t t t ... t t t r r r r r b

with x = unused, t = inner training, r = red = inner testing = hyperparameter tuning, b = outer testing

Then:

• The random uncertainty of the RMSE depends on the number of cases that enter the calculation of the RMSE: the more cases, the lower the variance on the RMSE estimate, thus the less noisy we expect the estimate to look.
So the blue estimates being based on 1 case each are expected to be more noisy than the red estimates which are based on 5 cases each.

• Unless you retrain after fixing hyperparameters on t t t ... t t t r r r r r (do you?), the blue predictions have more lag, i.e. are further into future. This may lead to higher variance error in the predictions, thus also of the blue error.

• Also, if the inner RMSE estimate is based on 5 consecutive time points and shifted one at a time, it must have considerable auto-correlation (which it does). Autocorrelation will make it look ness noisy.

• Due to the complex interaction (the inner RMSE estimte being actually part of the training) I find it difficult to say whether the outer RMSE estimate is surprisingly noisy. A simulation of the behaviour of the two RMSEs may be helpful.

As RMSE is the square root of the average squared error (as opposed to averaging RMSEs) the variance uncertainty of our RMSE estimate does not quite follow the simple variance of the mean $$s^2_\bar x = \frac{1}{n} s^2_x$$ dependency. See variance of variance estimate for more information.

As a side note, I had only a very quick glance at the blog post you linked, so I cannot quite comment on its total virtue. However, already the first glance did raise some red flags:

• RMSE over 1 case is just absolute error of that one case.
Of course, one can use this - but a reminder of this property to the unsuspecting reader would have been good. I wouldn't have mentioned this, though, if there weren't the nex concern:
• Taking the average of $$i$$ 1-case-"RMSE"s is not the same as calculating the RMSE over $$i$$ cases, it's the MAE of those $$i$$ cases. Again, it is not forbidden to do this - but it should IMHO be clearly labeled as "mean RMSE as opposed to RMSE of the pooled predictions", and again the unsuspecting reader should be made aware of what is done. Also a justification would be in place, IMHO.

Together, these two points make me suspicious whether the author actually calculates what they think they are calculating (to me this has a distinct "smell" of possible error with operator precedence).

• Looking at RMSE of models with a varying length of the training window makes a lot of sense. But pooling these results across widely varying training window sizes IMHO does require at the very least a justification.
I don't have access to the papers the blog references, but I'd have expected methods with rolling origin in their name to actually have a moving origin, i.e. a moving training window of fixed size. But the day-chaining described as being the same does use expanding windows with fixed origin (and moving end).

• "cross validation" seems to me a rather unlucky choice of name as it's explicitly about not doing cross validation due to the time dependency - though to be fair that may be the result of the blog post author trading off unlucky choice of name vs. term established by the Bergmeir & Benítez paper.
I'd have used expanding window validation (or rolling window valiation in your case) instead.

• Thank you so much for this very thorough explanation. Can you please see my answere below, I just want to show you the updated plot ?! – Perl Sep 17 '19 at 13:49
• Hiyam, there's nothing wrong with rolling (fixed window size) validation here - IMHO it is the method that makes most sense, in any case more than expanding windows, and cross validation (in the usual sense: test cases may be from in between or before training cases) does not make sense with time series! Your hyperparameter optimization including model choice can be done in one inner validation. A single time series with 150 cases will not allow you much hyperparameter optimization, though: you may be better off deciding for a low complexity model on the basis of your domain knowledge. – cbeleites unhappy with SX Sep 18 '19 at 8:05

@cbeleites Thank you so much!!! Indeed I realize what you are saying is very very true. The reason why I did not using rolling forecasts directly is because I have suite of models: Lasso, Ridge.., etc. from which I want to pick among them the best performing model, and then do rolling forecast using THAT picked model afterwards. However I want to tune their hyper parameters. That is why I am resorting to nested cross validation. Now after your comment, I am doing the following: n the outer loop, I divide the data now into 70 training and 20 testing, and the 70 training I divide them in the inner loop into 50 training and 10 validation. As I will use expanding window as you have suggested, the training sizes in both the inner and the outer loop increase one time step at a time, with the testing and validation sizes being fixed to 20 and 10 respectively in each of the folds (inner and outer)

This is what I got so far:

Now it is very clear, thanks to you, that doing "RMSE" on 1 sample was a very big, let me call it, mistake (since as you said it is just the absolute error).