I am trying to forecast revenue of a company, using neural networks. The response is a time series of monthly revenues from 11/2008 to 05/2016, and there are about 45 predictors (including lagged values of a unique predictors).
After dividing data into training (first 60 observations) and testing (rolling window of 3 months at a time, for a total of 30 months) sets, I have identified a model (an ensemble of a 5 neural networks and ETS), which yields MAPE below 3%. For identifying the best neural networks, I ran a macro, trying different seed and layer combinations, and chose 5 "best" with minimum MAPEs. (Is this something we can do?)
I need further assurance that the model is good, and was wondering if I could reverse the time series and corresponding predictors, fit neural network models with "best" seed-layer combinations identified before (with non-reversed time series), make predictions (for 2009 and 2008 data, using 2016-2010 data) and calculate errors. I am not sure if this procedure is correct, and would like to get some advice on it.