Time series cross validation by reversing the series

Question

I am trying to forecast revenue of a company, using neural networks. The response is a time series of monthly revenues from 11/2008 to 05/2016, and there are about 45 predictors (including lagged values of a unique predictors).

After dividing data into training (first 60 observations) and testing (rolling window of 3 months at a time, for a total of 30 months) sets, I have identified a model (an ensemble of a 5 neural networks and ETS), which yields MAPE below 3%. For identifying the best neural networks, I ran a macro, trying different seed and layer combinations, and chose 5 "best" with minimum MAPEs. (Is this something we can do?)

I need further assurance that the model is good, and was wondering if I could reverse the time series and corresponding predictors, fit neural network models with "best" seed-layer combinations identified before (with non-reversed time series), make predictions (for 2009 and 2008 data, using 2016-2010 data) and calculate errors. I am not sure if this procedure is correct, and would like to get some advice on it.

Answer 1

One would normally assume the data one intends to predict are generated by the same process that generated the sample data. As we train a model on the sample data, we fit some patterns. When forecasting, we extrapolate them into the future.

Once we reverse the time, the patterns change. The model that had captured the proper-time patterns well might be quite bad for capturing reversed-time patterns. Therefore, the reversed time series need not be relevant in terms of model training and performance evaluation intended for the proper time flow.

Take an example of an MA(1) process.

$$ x_t = \varepsilon_t + \theta_1 \varepsilon_{t-1}. $$

Suppose we have a sample generated by this process. If we consider the class of ARMA models, we might hope to discover correctly that the data generating process is MA(1). We might further hope to estimate the model parameters ($\theta_1$ and the error variance $\sigma^2$) accurately enough and successfully use the estimated model for forecasting. The point forecast from the estimated MA(1) model would be

$$ \hat{x}_{t|t-1} = \hat\theta_1 \hat\varepsilon_{t-1}. $$

Once we reverse the time, we have to express $x_t$ in terms of $x_{t+i}$ and $\varepsilon_{t+j}$ for $i,j>0$, estimate the model and use it for forecasting. The problem is, the true model in reversed time will not be an MA(1). Moreover, the MA(1) model might not even be a good approximation for the reversed series. In the class of ARMA models, there will exist numerous better approximations to the reversed process than the MA(1) model. For example, if you keep all your candidate models and test their performance on both the proper time series and the reversed one, you might see quite some changes in the ranking of performance.

Therefore, reversing time need not be helpful when you want to test your model performance. It could be helpful in some special cases (e.g. when the data is generated by a linear time trend plus random noise), but not necessarily in general.

Answer 2

As I wrote in the comments there's nothing inherently wrong with this approach per se. In fact, time separated holdout sample might be the only way you can cross validate the model.

If you had multiple companies, you could create a cross-validation sample from a subset of companies. However, it seems that you have only one company, then time separated sub sample is the only option for you.

You're dealing with 2008-2016 period which brings up its own issue: you have only a part of business cycle. Look at NBER's recession dates here. The last recession went from Dec 2007 to June 2009. So, if you try to predict the data from 2008-2009 based on data since 2010, you should expect poor results.

It's always good to have data spanning multiple business cycles to separate out the impact of the economic environment. It's like trying to do seasonal analysis when you have only half year of data: not going to work very well.

On top of macroeconomic life cycle, you also have the industry and firm life cycles. In different stages of life of a firm it goes through very different patterns of revenue growth and other things like capital investments. This all complicates the cross validation design.

One approach I may suggest is to use random non-continuous observations for the hold out. For instance, you take a few quarters in 2008-2009 + a few qurters outside this period. This way your training sample will include some observations from 2008-2009 too. This is a bit inconvenient for time series models such as ARIMA, but it can be done if you estimate the samples with state space methods. The last recession was very unique, therefore it's better to include its observations in any training sample

UPDATE:

As I commented in @RichardHardy's answer, I'm assuming that you're not actually trying to reverse time in a literal sense. You're not trying to "predict" January 2008 revenues based on data since Feb 2008 onwards. I'm assuming you're trying to train the model on 2010-2016 data, then apply the model to data in 2008. This approach is not only Ok, it's actually used in practice.