Is it valid to use bootstrapping to generate a confidence interval of an accuracy measured by rolling window analysis?

Question

I would like to generate confidence intervals for the accuracy of a model that I use to forecast time series data. In order to get an average accuracy for my model I use rolling window analysis (See under the heading - Rolling Window Analysis for Predictive Performance).

My model produces a multi-step forecast and due to a limited test set size I perform an overlapping window analysis to compute an average accuracy for the model. Let $T=1,2,3,4,5$ be my test data and $h=2$ be the forecast horizon. Then I have the following overlapping test sets

$$ T_1 = 1,2 \\ T_2 = 2,3 \\ T_3 = 3,4 \\ T_4 = 4,5. \\ $$

Each time the model produces a forecast it takes the first point from the previous test set and appends it to its input, as this is how the model would be used in practice. To demonstrate the input data for the above test sets would be (with an input window size of 3)

$$ I_1 = -2,-1,0 \\ I_2 = -1,0,1 \\ I_3 = 0, 1, 2 \\ I_4 = 1,2,3. $$

I compute an accuracy measure for my predictions and get an average accuracy of the model made over the test sets. I would like to compute a confidence interval that tells me how probable it is that the accuracy of my model is within a given range. I have found out that I can do this using bootstrapping but I am dubious as to whether it holds when my test sets are not independent from each other?

As I have understood it my data set would be the set of accuracies computed above for each model $A_{1:5}$. I can estimate the distribution $\delta$ of these accuracies by generating many samples using sample with replacement and using the following equation

$$\delta^*=\bar{x}^*-\bar{x}.$$

I repeat this many times in order to achieve a high precision and then simply order the results $\delta$s and select the quantiles I am interested in, let's say 0.1 and 0.9 to get an 80% confidence interval.

I would be comfortable doing this if my test sets were not overlapping and therefore independent:

$$ T_1=1,2 \\ T_2=3,4 $$

Here we have to lose the last point as it doesn't fit into our forecasting horizon. I would be fine with bootstrapping to compute a confidence interval if my test sets were as described just here. I am not certain I can do this if they overlap as described initially. Can I?

If I cannot and it is not always possible for me to generate independent test sets, due to lack of available data is it still possible to compute a confidence interval using another method?

Is there a minimum number of data points my empirical distribution has to have in order for it to be a valid approximation of the true distribution? For example if I had only enough data points to have one test set, then the empirical data distribution can only ever be that one number, so there would be no point in bootstrapping?

Answer 1

After thinking about the problem some more I was not able to satisfactorily answer whether or not the sampled accuracies from dependent test sets would be a valid way to generate confidence intervals for the accuracy of a model.

I did however come up with a different way of solving the problem which might be useful for someone in the future.

Instead of using rolling window analysis (as detailed in the post above and the link) you can simply take your time series and apply the bootstrap here. So let's say we have time series $T = [1:10]$ and a forecast horizon of two. We can generate $B$ bootstrapped datasets using a stationary bootstrap with a moving block size (optimally set to be $O(n^{\frac{1}{3}})$ where $n$ is the length of $T$) as our time series data is dependent.

This simplifies the problem as now we can simple use a simple out-of-sample method on each bootstrapped series. We can then compute the average accuracy on these out-of-sample test sets. With this method you will train $B$ models, each one on data that is assumed (due to the bootstrap) to share the same distribution as the original series $T$. This way we can see how well the model has learnt the actual 'information within the series' as this should be shared among all the bootstrapped data sets and our test sets on the different models will be independent.

As an example let's say $B=2$ (it should be much greater, but this is just an example)

We generate then bootstrapped series $B_1$ and $B_2$:

$B_1 = [2, 1, 3, 1, 2, 4, 2, 8, 9, 10]$ $B_2 = [4, 5, 6, 7, 5, 6, 8, 4, 3, 4]$

We can then split these series leaving the length of our forecast horizon $H$ for each one as an out-of-sample test.

$B_{\text{train}1} = [2, 1, 3, 1, 2, 4, 2, 8]$, $B_{\text{test}1} = [9, 10]$ $B_{\text{train}2} = [4, 5, 6, 7, 5, 6, 8, 4]$, $B_{\text{test}2} = [3, 4]$

We can then train two models $M_1$ and $M_2$ on $B_{\text{train}1}$ and $B_{\text{train}2}$ respectively and compute an accuracy against the corresponding test set.

We can then compute the standard error or any other statistic that we would like to from this population of samples and find a mean or standard deviation for example.