# How to correctly compare the accuracy of different forecasting methods using bootstrapping with time series forecasting

I am currently working on a forecasting project and I have tried several different models to forecast with. Having trained and tuned my models I want to pick which model works best for each time series that I have.

My process thus far (due to limited length of the time series) has been to split my data into a train and test set where my test set consists of 1 complete forecast horizon.

I then further split my train set up to allow me to tune my models by adjusting suitable parameters. The models I have used are ARIMA, Exponential smoothing and an LSTM. The process I used to adjust the parameters was by taking my training data and framing my problem as a supervised learning problem and computing an accuracy measure via walk forward validation. I'll illustrate with an example.

Let $$T = 1,\cdots,13$$ be my time series data and $$h=3$$ be my forecast horizon. I remove the last $$h$$ points from $$T$$ and keep this as an untouched test set, $$T_{\text{test}}=11, 12, 13$$ with the remaining points being left as a training set $$T_{\text{train}}=1, \cdots, 10$$.

I then train my model $$M$$ using walk forward validation to find suitable parameters :

Train 1: [1, 2, 3, 4 | 5, 6, 7] compute accuracy
Train 2: [2, 3, 4, 5 | 6, 7, 8] compute accuracy
Train 3: [3, 4, 5, 6 | 7, 8, 9] compute accuracy
Train 4: [4, 5, 6, 7 | 8, 9, 10] compute accuracy


I then took an average of all the measured accuracies to determine an average accuracy for that model with those parameters. I repeated this for many different variations of parameters, finally selecting the parameters which gave me the best accuracy on average.

I used this process for each of my models. I then took each model and re-trained it on the whole training dataset using the parameters I found to be best for each model. I would now like to take each model and compute predictions so that I can calculate a final accuracy measure for each model and select the most accurate one.

As I only have a test set that is equal to the size of my forecast horizon I cannot make several rolling predictions and evaluate an average accuracy of each model. I can only make one set of predictions. This is a limitation of my dataset and it is not something that I can change. I would like to know how can I meaningfully say that model A is more or less accurate than model B?

As an example let's say that model A scored 80% accuracy against the test set and model B scored 75% accuracy. As far as I can see I can only really say that model A is 5% more accurate at predicting this test set than model B. I would like to know if there is a way I can compute any uncertainty so I can make a more meaningful comparison and decision about which model is better?

Something like model A scored 80% $$\pm$$ 2% accuracy and model B scored 75% $$\pm$$ 6% accuracy, therefore model A is preferable for forecasting time series $$T$$. I would also like to be able to provide some kind of confidence rating for my model, so I would be able to say something like "model A will predict 99% of the time to 80% accuracy" or something along those lines. I am sure these things must be possible to compute but I am not really sure how given the small size of my data left over for testing.

The other option I suppose would be to keep a larger dataset aside for testing and have a much smaller set of data points left over to tune my model parameters. Which would allow me to compute maybe one to three overlapping rolling forecasts against the test set.

EDIT

So I have been investigating and I have discovered a technique called bootstrapping which I think I can apply to my problem in order to gain to assess the uncertainty of the accuracy of my model. But not how probable the prediction is to be correct, is this correct? I have also come across the term "conformal prediction" but I am as yet still unclear as to how this applies specifically to my problem although I am sure it does.

The method of bootstrapping as applied to time series forecasting however has me a bit stumped. I have understood the concept of creating bootstrapped datasets to create new datasets to fit my model to and therefore come up with a distribution of predictions but I am unsure how to apply this in practice.

I would like first to confirm that I should bootstrap after I have finished tuning my model to make a final assessment of the accuracy of the model, not during the tuning phase?

I have seen that I can compute the bootstrapped error using the following formula taken from the elements of statistical learning page 251:

$$\hat{Err}^{(.632)}=.368.\bar{\text{err}}+.632.\hat{Err}^{(1)} \\ \hat{Err}^{(1)}=\frac{1}{N}\sum^N_{i=1}\frac{1}{|C^{-i}|}\sum_{b\in C^{-i}}L(y_i, \hat{f}^{*b}(x_i)),\\$$ where $$C^{-i}$$ is the set of indices of the bootstrap samples $$b$$ that do not contain observation $$i$$ and $$|C^{-i}|$$ is the number of samples. There are several things I do not understand in this formula, the first and probably the most obvious is, what is $$\bar{\text{err}}$$? Is it just the average error I computed from all the predictions that model made?

Secondly will I need to make any variations to how this bootstrapped error is computed if I increase the size of my test set so that I can take a few rolling horizons and compute an average accuracy? For example if my forecast horizon is 2 points, instead of holding 2 points back as a test set holding instead 6 and being able to compute an accuracy for each test set and take an average $$[1,2], [2,3], [3,4], [4,5], [5,6]$$.

Lastly are the any things I should be aware of that are common in a typical bootstrapping scenario but which are not valid to do for the case of time series forecasting and regression problems?