0
$\begingroup$

I have implemented a leave one out cross validation to calculate errors between daily forecast and observed values for spatio-temporal data taken in a given season (summer say). I have further implemented a vector stationary block bootstrap (SBB) to account for the spatial and temporal correlation to calculate RMSE and the corresponding confidence intervals (CIs). I am also comparing multiple forecast methods (m) so I repeat the above procedure for each method.

To further complicate things, I have data for the same time period (season) for n years so I actually have n RMSE and CI values for each forecast method.

What is the best approach to get an overall RMSE (over all years) for each method?

(Note, the time periods from one year to the next are not contiguous so the bootstrap can only be applied within each year)

$\endgroup$
1
  • $\begingroup$ If you implemented vector stationary block bootstrap because LOOCV had obvious drawbacks, why would you use LOOCV at all? $\endgroup$
    – Firebug
    Commented Sep 22, 2021 at 16:26

1 Answer 1

0
$\begingroup$

This is more a suggestion than an answer ATM, but too long for a comment.

Intuitively, this seems somewhat similar to obtaining multiple results from different cross validation partitions and repeats$^1$. Therefore, I would try calculating statistics across your different RMSE and CI results in similar manner, then compare different models/select a suitable model based on those. Statistics usually taken into account would e.g. be a mean/median, sd/mad - or one boxplot per model ordered by median - which together give you an idea about the average performance and performance spread for each model. This should then help in selecting models/model parametrizations to take a closer look at.

$^1$ The difference is that changes in error over time could occur with your models and data, which is not addressed with looking at results from classic CV partition and repeats. You might want to think about a way of adding this information in your metrics in some way.

$\endgroup$
1
  • $\begingroup$ Just to be clear, are you saying to make a boxplot of the yearly RMSE values for each method? How would I take the CI's of each RMSE value into account? $\endgroup$
    – piyushnz
    Commented Jun 23, 2016 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.