2
$\begingroup$

I am trying to forecast revenue of a company, using neural networks. The response is a time series of monthly revenues from 11/2008 to 05/2016, and there are about 45 predictors (including lagged values of a unique predictors).

After dividing data into training (first 60 observations) and testing (rolling window of 3 months at a time, for a total of 30 months) sets, I have identified a model (an ensemble of a 5 neural networks and ETS), which yields MAPE below 3%. For identifying the best neural networks, I ran a macro, trying different seed and layer combinations, and chose 5 "best" with minimum MAPEs. (Is this something we can do?)

I need further assurance that the model is good, and was wondering if I could reverse the time series and corresponding predictors, fit neural network models with "best" seed-layer combinations identified before (with non-reversed time series), make predictions (for 2009 and 2008 data, using 2016-2010 data) and calculate errors. I am not sure if this procedure is correct, and would like to get some advice on it.

$\endgroup$
  • $\begingroup$ Nothing inherently wrong with this. $\endgroup$ – Aksakal Jul 11 '16 at 14:05
  • $\begingroup$ @Aksakal, but is there anything "right" with this? WIll the outcome of the exercise be relevant and/or informative with respect to the real modelling goal? $\endgroup$ – Richard Hardy Jul 11 '16 at 14:08
  • $\begingroup$ On this site there's no need to say "thank you" at the end of your post - it might seem rude at first, but it's part of the philosophy of this site (tour) to "Ask questions, get answers, no distractions" and it means future readers of your question don't need to read through the pleasantries. $\endgroup$ – Silverfish Jul 11 '16 at 14:15
  • $\begingroup$ When I did the reversing and prediction, I got very high errors. On one hand, I have this performing great on the recent 30 months data, and on the other hand, it doesn't do well on the reversed data. Is this a good enough reason to reconsider my model, or should I still use it, considering my aim is predicting revenues for the next 3 months? $\endgroup$ – Rahul Jul 11 '16 at 14:23
  • $\begingroup$ @RichardHardy, I think OP had doubts whether what he calls reverse cross validation is allowed at all. He'll face all the usual issues that come with time separated hold out samples, e.g. how relevant is the data in one period to the other? Given that 2008/2009 is at the bottom of the Great Recession, it's a really big question. However, it's nothing to do with reversing the time. $\endgroup$ – Aksakal Jul 11 '16 at 14:44
1
$\begingroup$

One would normally assume the data one intends to predict are generated by the same process that generated the sample data. As we train a model on the sample data, we fit some patterns. When forecasting, we extrapolate them into the future.

Once we reverse the time, the patterns change. The model that had captured the proper-time patterns well might be quite bad for capturing reversed-time patterns. Therefore, the reversed time series need not be relevant in terms of model training and performance evaluation intended for the proper time flow.

Take an example of an MA(1) process.

$$ x_t = \varepsilon_t + \theta_1 \varepsilon_{t-1}. $$

Suppose we have a sample generated by this process. If we consider the class of ARMA models, we might hope to discover correctly that the data generating process is MA(1). We might further hope to estimate the model parameters ($\theta_1$ and the error variance $\sigma^2$) accurately enough and successfully use the estimated model for forecasting. The point forecast from the estimated MA(1) model would be

$$ \hat{x}_{t|t-1} = \hat\theta_1 \hat\varepsilon_{t-1}. $$

Once we reverse the time, we have to express $x_t$ in terms of $x_{t+i}$ and $\varepsilon_{t+j}$ for $i,j>0$, estimate the model and use it for forecasting. The problem is, the true model in reversed time will not be an MA(1). Moreover, the MA(1) model might not even be a good approximation for the reversed series. In the class of ARMA models, there will exist numerous better approximations to the reversed process than the MA(1) model. For example, if you keep all your candidate models and test their performance on both the proper time series and the reversed one, you might see quite some changes in the ranking of performance.

Therefore, reversing time need not be helpful when you want to test your model performance. It could be helpful in some special cases (e.g. when the data is generated by a linear time trend plus random noise), but not necessarily in general.

$\endgroup$
  • $\begingroup$ I think you're misinterpreting the reverse word in OP's question here: "reverse the time, we have to express $x_t$ in terms of $x_{t+i}$." I think what @Rahul means is that he'll estimate the model as usually on the training sample from 2010-2016, then apply it to holdout (cross validation) sample from 2008-2009. For instance, if he's building MA(1) model on monthly revenues, he'll seed with January 2008 revenue, then will produce 23 months of forecasts to compare with actuals in Feb 2008-Dec 2009 using trained model. He's not going to actually reverse time in the sense you describe it. $\endgroup$ – Aksakal Jul 11 '16 at 20:31
1
$\begingroup$

As I wrote in the comments there's nothing inherently wrong with this approach per se. In fact, time separated holdout sample might be the only way you can cross validate the model.

If you had multiple companies, you could create a cross-validation sample from a subset of companies. However, it seems that you have only one company, then time separated sub sample is the only option for you.

You're dealing with 2008-2016 period which brings up its own issue: you have only a part of business cycle. Look at NBER's recession dates here. The last recession went from Dec 2007 to June 2009. So, if you try to predict the data from 2008-2009 based on data since 2010, you should expect poor results.

It's always good to have data spanning multiple business cycles to separate out the impact of the economic environment. It's like trying to do seasonal analysis when you have only half year of data: not going to work very well.

On top of macroeconomic life cycle, you also have the industry and firm life cycles. In different stages of life of a firm it goes through very different patterns of revenue growth and other things like capital investments. This all complicates the cross validation design.

One approach I may suggest is to use random non-continuous observations for the hold out. For instance, you take a few quarters in 2008-2009 + a few qurters outside this period. This way your training sample will include some observations from 2008-2009 too. This is a bit inconvenient for time series models such as ARIMA, but it can be done if you estimate the samples with state space methods. The last recession was very unique, therefore it's better to include its observations in any training sample

UPDATE:

As I commented in @RichardHardy's answer, I'm assuming that you're not actually trying to reverse time in a literal sense. You're not trying to "predict" January 2008 revenues based on data since Feb 2008 onwards. I'm assuming you're trying to train the model on 2010-2016 data, then apply the model to data in 2008. This approach is not only Ok, it's actually used in practice.

$\endgroup$
  • $\begingroup$ Very little of this addresses the original question (although probably a lot of it could be useful in general). Regarding the original question, reversing time means changing the patterns that the model is supposed to learn. Therefore, using reversed-time data must be roughly as useful as using randomly generated data -- that is, not useful. $\endgroup$ – Richard Hardy Jul 11 '16 at 16:37
  • $\begingroup$ @RichardHardy, " changing the patterns that the model is supposed to learn" a very generic vague statement. Elaborate if you want to be useful $\endgroup$ – Aksakal Jul 11 '16 at 17:03
  • $\begingroup$ I posted an answer where I elaborate on what I mean. $\endgroup$ – Richard Hardy Jul 11 '16 at 20:15
  • $\begingroup$ I am really sorry for not mentioning clearly. I was actually planning to reverse in literal sense. I should have written 2016-2010. I just updated it. I believe now I know what is acceptable and what is not. In short, reversing the entire sequence would not be meaningful because of reasons given by @RichardHardy, and holding out 2 years could have been meaningful if not for recession. Both the facts are really helpful! $\endgroup$ – Rahul Jul 11 '16 at 20:57
  • $\begingroup$ @Rahul, I'm curious how were you planning to reverse "lagged values of predictors"? $\endgroup$ – Aksakal Jul 11 '16 at 21:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.