16
$\begingroup$

I would like to build an algorithm that would be able to analyze any time series and "automatically" choose the best traditional/statiscal forecasting method (and its parameters) for the analyzed time series data.

Would it be possible to do something like this? If yes, can you give me some tips on how can this be approached?

$\endgroup$
  • 3
    $\begingroup$ No, this cannot reasonably be achieved. Frequently, there is not enough data to distinguish between two reasonable models, never mind all possible models. Achieving a best model would be require that the physics be known in absolute terms, and very frequently the modelling assumptions are not even known, and/or are untested/untestable. $\endgroup$ – Carl Dec 6 '18 at 5:29
  • 3
    $\begingroup$ No. There is no way to determine which model is the best. Python is not relevant in this discussion. Nevertheless there are attempts with good results. For example github.com/facebook/prophet project. It also has Python binding. $\endgroup$ – Cagdas Ozgenc Dec 6 '18 at 6:10
  • 3
    $\begingroup$ I am voting to leave open because I think it's a reasonable question - even if the answer is "no". I'd suggest removing python from the title, because it's not relevant or especially on topic here. $\endgroup$ – mkt - Reinstate Monica Dec 6 '18 at 10:22
  • 1
    $\begingroup$ I have removed python from the title as suggested. Thank you for your answers. $\endgroup$ – StatsNewbie123 Dec 6 '18 at 14:29
  • 2
    $\begingroup$ See "no free lunch" theorem. $\endgroup$ – AdamO Dec 6 '18 at 14:47
20
$\begingroup$

First you need to note that the approach outlined by IrishStat is specific to ARIMA models, not to any generic set of models.

To answer your main question "Is it possible to automate time series forecasting?":

Yes it is. In my field of demand forecasting, most commercial forecasting packages do so. Several open source packages do so as well, most notably Rob Hyndman's auto.arima() (automated ARIMA forecasting) and ETS() (automated exponential smoothing forecasting) functions from the open source Forecast package in R see here for details on these two functions. There's also a Python implementation of auto.arima called Pyramid, although in my experience it is not as mature as the R packages.

Both the commercial products that I mentioned and the open source packages I mentioned work based on the idea of using information criteria to choose the best forecast: You fit a bunch of models, and then select the model with the lowest AIC, BIC, AICc, etc....(typically this is done in lieu of out of sample validation).

There is however a major caveat: all of these methods work within a single family of models. They choose the best possible model amongst a set of ARIMA models, or the best possible model amongst a set of exponential smoothing models.

It is much more challenging to do so if you want to choose from different families of models, for example if you want to choose the best model from ARIMA, Exponential smoothing and the Theta method. In theory, you can do so in the same way that you do within a single family of models, i.e. by using information criteria. However in practice, you need to calculate the AIC or BIC in exactly the same way for all models considered, and that is a significant challenge. It might be better to use time series cross-validation, or out of sample validation instead of information criteria, but that will be much more computationally intensive (and tedious to code).

Facebook's Prophet package also automates forecast generation based on General Additive Models See here for details. However Prophet fits only one single model, albeit a very flexible model with many parameters. Prophet's implicit assumption is that a GAM is "the one model to rule them all", which might not be theoretically justified but is very pragmatic and useful for real world scenarios.

Another caveat that applies to all of the above mentioned methods: Presumably you want to do automated time series forecasting because you want to forecast multiple time series, too many to analyze manually. Otherwise you could just do your own experiments and find the best model on your own. You need to keep in mind that an automated forecasting approach is never going to find the best model for each and every time series - it is going to give a reasonably good model on average over all the time series, but it is still possible that some of those time series will have better models than the ones selected by the automated method. See this post for an example of this. To put it simply, if you are going to go with automated forecasting - you will have to tolerate "good enough" forecasts instead of the best possible forecasts for each time series.

$\endgroup$
  • $\begingroup$ Thank you for such an insightful answer. Honestly my initial idea when asking this question was to be able to choose from different families of models. $\endgroup$ – StatsNewbie123 Dec 6 '18 at 17:05
  • $\begingroup$ What if the algorithm was to be applied to a more specific domain, like business forecasting (for production planning and inventory management for example)? Do you think that it could achieve better results? $\endgroup$ – StatsNewbie123 Dec 6 '18 at 17:13
  • 2
    $\begingroup$ @JoanaClaudino yes that would be easier - that is the gist of what commercial demand forecasting packages do. But it is easier because typically when you limit yourself to one specific business domain you can limit yourself to one specific family of models (time series coming from the same business domain will have similar characteristics in terms of seasonalities, data densities, etc....hence it is safe to assume that one family will be suitable for all of them ) $\endgroup$ – Reinstate Monica Dec 6 '18 at 17:18
  • $\begingroup$ Okay, I get it now. Thank you once again for such a great help. $\endgroup$ – StatsNewbie123 Dec 6 '18 at 17:24
11
$\begingroup$

My suggested approach encompasses models that are much more general than ARIMA as they include the potential for seasonal dummies that may change over time , multiple levels ,multiple trends , parameters that may change over time and even error variances that may change over time. This family is more precisely called ARMAX models but for complete transparency does exclude a (rare) variant that has multiplicative structure.

You asked for tips and I believe that this might be a good one to get you started.

I would suggest that you write code to follow/emulate this flowchart/workflow. The "best model" could be found by evaluating the criterion that you specify ... it could be the MSE/AIC of the fitted data or it could be the MAPE/SMAPE of withheld data or any criterion of your choice.

Be aware as the detailing of each of these steps can be quite simple if you are unaware of some of the specific requirements/objectives/constraints of time series analysis BUT it can be (should be !) more complex if you have a deeper understanding/learning/appreciation of the complexities/opportunities present in thorough time series analysis.

enter image description here

enter image description here

I have been asked to provide further direction as to how one should go about automating time series modelling ( or modelling in general ) https://stats.stackexchange.com/search?q=peeling+an+onion contains some of my guidance on "peeling onions" and related tasks .

AUTOBOX actually details and shows the interim steps as it forms a useful model and could be a useful teacher in this regard. The whole scientific idea is to "add what appears to be needed" and "delete what appears to be less than useful" . This is the iterative process suggested by Box and Bacon in earlier times.

Models need to be complex enough (fancy enough) but not too complex (fancy). Assuming that simple methods work with complex problems is not consistent with scientific method following Roger Bacon and tons of followers of Bacon. As Roger Bacon once said and I have often paraphrased : To do science is to search for repeated patterns. To detect anomalies is to identify values that do not follow repeated patterns. For whoever knows the ways of Nature will more easily notice her deviations and, on the other hand, whoever knows her deviations will more accurately describe her ways. One learns the rules by observing when the current rules fail.In the spirt pf Bacon by identifying when the currently identified "best model/theory" is inadeqaute one can then iterate to "a better representation"

In my words "Tukey proposed Exploratory Data Analysis (EDA) which suggested schemes of model refinement based upon evidented model deficiency suggested by the data" . This is the heart of AUTOBOX and of science. EDA is for seeing what the data can tell us beyond the formal modeling or hypothesis testing task.

The litmus test of an automatic modelling program is quite simple . Does it separate signal and noise without over-fitting ? Empirical evidence suggests that this can and has been done. Forecasting accuracies are often misleading because the future is not accountable for the past and depending on which origin you pick results can and do vary.

$\endgroup$
  • $\begingroup$ Thank you so much. Your answer is extremely helpful! $\endgroup$ – StatsNewbie123 Dec 6 '18 at 14:14
  • $\begingroup$ Thank you, IrishStat. This is insightful. Are we ever justified to keep a parameter as opposed to "delete parameter", despite the parameter not being statistically significant? $\endgroup$ – ColorStatistics Dec 6 '18 at 15:12
  • 1
    $\begingroup$ Yes. if it gives you a more "pleasing answer". A slope coefficient that is not different from 0.0 may provide a more realistic (if not-statistically significant) forecast, Otherwise carrying excess coefficients is illogical and causing increased uncertainty in the variance forecast. Some authors unable to deal with tests of necessity and sufficiency blindly/naively assume that model identification is a 1 step process while Box and Jenkins ( and everybody else !) are concerned with making the model as complex as needed but not overly complex ..read Albert E. , J.W Tukey et al for more on this. $\endgroup$ – IrishStat Dec 6 '18 at 15:57
  • 3
    $\begingroup$ "This suggested automated forecasting approach is going to find the best model for each and every time series by tailoring the form of the model i.e. customizing it much like a customized suit." That's a pretty strong statement - if accurate, it means this approach would win the M3, M4 competitions hands down. Has it? $\endgroup$ – Reinstate Monica Dec 7 '18 at 0:05
  • 1
    $\begingroup$ @joana-claudino I am particularly interested in your ISEG project as I am also interested in evolving a Python version of my suggested procedures. This of course would be off site from SE. $\endgroup$ – IrishStat Dec 7 '18 at 16:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.